> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade a cluster in Kubernetes

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

This page shows how to upgrade a CockroachDB cluster that is <InternalLink path="deploy-cockroachdb-with-kubernetes">deployed on a Kubernetes cluster</InternalLink>.

This page is for Kubernetes deployments that are not using the CockroachDB operator. For guidance specific to the CockroachDB operator, read <InternalLink path="upgrade-cockroachdb-operator">Upgrade a Cluster in Kubernetes with the CockroachDB operator</InternalLink>.

New deployments of CockroachDB on Kubernetes are recommended to use the newer, fully-featured CockroachDB operator that's easier to deploy and supports scaling of multi-region clusters. To migrate an existing deployment to use the CockroachDB operator, read the <InternalLink path="migrate-cockroachdb-kubernetes-helm">Helm</InternalLink> and <InternalLink path="migrate-cockroachdb-kubernetes-operator">Public operator</InternalLink> migration guides.

## Overview

### Types of upgrades

* **Major-version upgrades**: A <InternalLink version="releases" path="index#major-versions">major-version upgrade</InternalLink>, such as from v24.2 to v24.3, may include new features, updates to cluster setting defaults, and backward-incompatible changes. Performing a major-version upgrade requires an additional step to finalize the upgrade.

  As of 2024, every second major version is an <InternalLink version="releases" path="release-support-policy#innovation-releases">Innovation release</InternalLink>. Innovation releases offer shorter support windows and can be skipped.
* **Patch upgrades**: A <InternalLink version="releases" path="patch-releases">patch upgrade</InternalLink> moves a cluster from one patch release to another within a major version, such as from v24.2.3 to v24.2.4. Patch upgrades do not introduce backward-incompatible changes.

  A major version of CockroachDB has two phases of patch releases: a series of **testing releases** (beta, alpha, and RC releases) followed by a series of **production releases** (vX.Y.0, vX.Y.1, and so on). A major version’s first production release (the.0 release) is also known as its GA release.

To learn more about CockroachDB major versions and patches, refer to <InternalLink version="releases" path="index">CockroachDB Releases Overview</InternalLink>.

### Compatible versions

A cluster may always be upgraded to the next major release. Every second major version is an Innovation release that can be deployed or skipped:

* If your cluster is running a major version that is a Regular release, it can be upgraded to either:
  * the next major version (an Innovation release)
  * the release that follows the next major version (the next Regular release, once it is available, skipping the Innovation release).
* If a cluster is running a major version that is labeled an Innovation release, it can be upgraded only to the next Regular release.

  <img src="https://mintlify.s3.us-west-1.amazonaws.com/cockroachlabs/docs/v26.1/images/common/version-skipping-diagram.png" alt="Diagram of CockroachDB major version upgrade availability, i.e. the ability to skip innovation releases" />

On Kubernetes, the upgrade is a [staged update](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set#staging-an-update) in which each pod's container image for CockroachDB is updated in a rolling fashion. The cluster remains available during the upgrade.

Select the cluster's deployment method to continue.

## Before you begin

<Tip>
  If you <InternalLink path="deploy-cockroachdb-with-kubernetes-openshift">deployed CockroachDB on Red Hat OpenShift</InternalLink>, substitute `kubectl` with `oc` in the following commands.
</Tip>

<Note>
  All `kubectl` steps should be performed in the <InternalLink path="deploy-cockroachdb-with-kubernetes#install-the-operator">namespace where you installed the operator</InternalLink>. By default, this is `cockroach-operator-system`.
</Note>

Before beginning a major-version or patch upgrade:

1. Verify the overall health of your cluster using the <InternalLink path="ui-cluster-overview-page">DB Console</InternalLink>:
   * Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as `SUSPECT` or `DEAD`, identify why the nodes are offline and either restart them or <InternalLink path="node-shutdown?filters=decommission#remove-nodes">decommission</InternalLink> them before beginning your upgrade. If there are `DEAD` and non-decommissioned nodes in your cluster, the upgrade cannot be finalized. If any node is not fully decommissioned, try the following:
     1. First, reissue the <InternalLink path="node-shutdown?filters=decommission#decommission-the-node">decommission command</InternalLink>. The second command typically succeeds within a few minutes.
     2. If the second decommission command does not succeed, <InternalLink path="node-shutdown?filters=decommission#recommission-nodes">recommission</InternalLink> and then decommission it again. Before continuing the upgrade, the node must be marked as `decommissioned`.
   * Under **Replication Status**, make sure there are `0` under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade.
   * In the **Node List**, make sure all nodes are on the same version. Upgrade them to the cluster's current version before continuing. If any nodes are behind, this also indicates that the previous major-version upgrade may not be finalized.
   * In the **Metrics** dashboards, make sure <InternalLink path="common-issues-to-monitor#cpu-usage">CPU</InternalLink>, <InternalLink path="common-issues-to-monitor#database-memory-usage">memory</InternalLink>, and <InternalLink path="common-issues-to-monitor#storage-capacity">storage</InternalLink> capacity are within acceptable values for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. If any of these metrics is above healthy limits, consider <InternalLink path="cockroach-start">adding nodes</InternalLink> to your cluster before beginning your upgrade.
2. Make sure your cluster is behind a <InternalLink path="recommended-production-settings#load-balancing">load balancer</InternalLink>, or your clients are configured to talk to multiple nodes. If your application communicates with only a single node, stopping that node to upgrade its CockroachDB binary will cause your application to fail.
3. By default, the <InternalLink path="architecture/storage-layer">storage engine</InternalLink> uses a compaction concurrency of 3. If you have sufficient IOPS and CPU headroom, you can consider increasing this setting via the `COCKROACH_COMPACTION_CONCURRENCY` environment variable. This may help to reshape the LSM more quickly in <InternalLink path="architecture/storage-layer#inverted-lsms">inverted LSM</InternalLink> scenarios; and it can lead to increased overall performance for some workloads. Cockroach Labs **strongly recommends** testing your workload against non-default values of this setting.
4. CockroachDB is designed with high fault tolerance. However, taking regular backups of your data is an operational best practice for <InternalLink path="disaster-recovery-planning">disaster recovery</InternalLink> planning. Refer to <InternalLink path="restoring-backups-across-versions#support-for-restoring-backups-into-a-newer-version">Restoring backups across versions</InternalLink>.
5. Review the <InternalLink version="releases" path="v26.1">v26.1 Release Notes</InternalLink>, as well as the release notes for any skipped major version. Pay careful attention to the sections for backward-incompatible changes, deprecations, changes to default cluster settings, and features that are not available until the upgrade is finalized.
6. Optionally [disable auto-finalization](#disable-auto-finalization) to preserve the ability to roll back a major-version upgrade instead of finalizing it. If auto-finalization is disabled, a major-version upgrade is not complete until it is finalized.

### Ensure you have a valid license key

To perform major version upgrades, you must have <InternalLink path="licensing-faqs#obtain-a-license">a valid license key</InternalLink>.

Patch version upgrades can be performed without a valid license key, with the following limitations:

* The cluster will run without limitations for a specified grace period. During that time, alerts are displayed that the cluster needs a valid license key. For more information, refer to the <InternalLink path="licensing-faqs">Licensing FAQs</InternalLink>.
* The cluster is <InternalLink path="licensing-faqs">throttled</InternalLink> at the end of the grace period if no <InternalLink path="licensing-faqs#set-a-license">valid license key is added to the cluster</InternalLink> before then.

If you have an **Enterprise Free** or **Enterprise Trial** license, you must enable telemetry using the <InternalLink path="cluster-settings">`diagnostics.reporting.enabled`</InternalLink> cluster setting, as shown below in order to finalize a major version upgrade:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING diagnostics.reporting.enabled = true;
```

If a cluster with an **Enterprise Free** or **Enterprise Trial** license is upgraded across patch versions and does not meet telemetry requirements:

* The cluster will run without limitations for a 7-day grace period. During that time, alerts are displayed that the cluster needs to send telemetry.
* The cluster is <InternalLink path="licensing-faqs">throttled</InternalLink> if telemetry is not received before the end of the grace period.

For more information, refer to the <InternalLink path="licensing-faqs">Licensing FAQs</InternalLink>.

If you want to stay on the previous version, you can roll back the upgrade before finalization.

## Perform a patch upgrade

To upgrade from one patch release to another within the same major version, perform the following steps on one node at a time:

1. Change the container image in the custom resource:

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   image:
     name: cockroachdb/cockroach:v24.3.29
   ```
2. Apply the new settings to the cluster:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   kubectl apply -f example.yaml
   ```

   The Operator will perform the staged update.
3. To check the status of the rolling upgrade, run `kubectl get pods`.
4. Verify that all pods have been upgraded:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl get pods \
   -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
   ```

   You can also check the CockroachDB version of each node in the <InternalLink path="ui-cluster-overview-page#node-details">DB Console</InternalLink>.

### Roll back a patch upgrade

To roll back a patch upgrade, repeat the steps in [Perform a patch upgrade](#perform-a-patch-upgrade), but configure the container image for the pods to the previous major version.

## Perform a major-version upgrade

To perform a major upgrade:

1. Change the container image image in the custom resource:

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   image:
     name: cockroachdb/cockroach:v26.1.1
   ```
2. Apply the new settings to the cluster:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   kubectl apply -f example.yaml
   ```

   The Operator will perform the staged update.
3. To check the status of the rolling upgrade, run `kubectl get pods`.
4. Verify that all pods have been upgraded:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl get pods \
   -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
   ```

   You can also check the CockroachDB version of each node in the <InternalLink path="ui-cluster-overview-page#node-details">DB Console</InternalLink>.
5. Before beginning a major-version upgrade, the Operator disables auto-finalization by setting the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version. Before finalizing an upgrade, follow your organization's testing procedures to decide whether to [finalize](#finalize-a-major-version-upgrade-manually) or [roll back](#roll-back-a-major-version-upgrade) the upgrade. After finalization begins, you can no longer roll back to the cluster's previous major version.

### Finalize a major-version upgrade manually

To finalize a major-version upgrade:

1. Connect to the cluster using the SQL shell:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=cockroachdb-public
   ```
2. Run the following command:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > RESET CLUSTER SETTING cluster.preserve_downgrade_option;
   ```

   A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return the previous version\`.

   You can monitor the process of the migration in the DB Console <InternalLink path="ui-jobs-page">Jobs page</InternalLink>. Migration jobs have names in the format `26.1-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs.

   The amount of time required for finalization depends on the amount of data in the cluster, because finalization runs various internal maintenance and migration tasks. During this time, the cluster will experience a small amount of additional load.

   When all migration jobs have completed, the upgrade is complete.
3. To confirm that finalization has completed, check the cluster version:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > SHOW CLUSTER SETTING version;
   ```

   If the cluster continues to report that it is on the previous version, finalization has not completed. If auto-finalization is enabled but finalization has not completed, check for the existence of <InternalLink path="node-shutdown?filters=decommission#status-change">decommissioning nodes</InternalLink> where decommission has stalled. In most cases, issuing the `decommission` command again resolves the issue. If you have trouble upgrading, [contact Support](https://cockroachlabs.com/support/hc).

### Roll back a major-version upgrade

To roll back to the previous major version before an upgrade is finalized:

1. Change the container image in the custom resource to use the previous major version:

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   image:
     name: cockroachdb/cockroach:v26.1
   ```
2. Apply the new settings to the cluster:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   kubectl apply -f example.yaml
   ```

   The Operator will perform the staged rollback.
3. To check the status of the rollback, run `kubectl get pods`.
4. Verify that all pods have been rolled back:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl get pods \
   -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
   ```

Rollbacks do not require finalization.

## Disable auto-finalization

For clusters managed by the Public operator, auto-finalization is disabled and cannot be enabled. A major version upgrade is not complete until it is manually [finalized](#finalize-a-major-version-upgrade-manually). The Public operator does not support the <InternalLink path="cluster-settings">cluster setting</InternalLink> `cluster.auto_upgrade.enabled`.

## Before you begin

If you <InternalLink path="deploy-cockroachdb-with-kubernetes-openshift">deployed CockroachDB on Red Hat OpenShift</InternalLink>, substitute `kubectl` with `oc` in the following commands.

All `kubectl` steps should be performed in the <InternalLink path="deploy-cockroachdb-with-kubernetes#install-the-operator">namespace where you installed the operator</InternalLink>. By default, this is `cockroach-operator-system`.

Before beginning a major-version or patch upgrade:

1. Verify the overall health of your cluster using the <InternalLink path="ui-cluster-overview-page">DB Console</InternalLink>:
   * Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as `SUSPECT` or `DEAD`, identify why the nodes are offline and either restart them or <InternalLink path="node-shutdown?filters=decommission#remove-nodes">decommission</InternalLink> them before beginning your upgrade. If there are `DEAD` and non-decommissioned nodes in your cluster, the upgrade cannot be finalized. If any node is not fully decommissioned, try the following:
     1. First, reissue the <InternalLink path="node-shutdown?filters=decommission#decommission-the-node">decommission command</InternalLink>. The second command typically succeeds within a few minutes.
     2. If the second decommission command does not succeed, <InternalLink path="node-shutdown?filters=decommission#recommission-nodes">recommission</InternalLink> and then decommission it again. Before continuing the upgrade, the node must be marked as `decommissioned`.
   * Under **Replication Status**, make sure there are `0` under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade.
   * In the **Node List**, make sure all nodes are on the same version. Upgrade them to the cluster's current version before continuing. If any nodes are behind, this also indicates that the previous major-version upgrade may not be finalized.
   * In the **Metrics** dashboards, make sure <InternalLink path="common-issues-to-monitor#cpu-usage">CPU</InternalLink>, <InternalLink path="common-issues-to-monitor#database-memory-usage">memory</InternalLink>, and <InternalLink path="common-issues-to-monitor#storage-capacity">storage</InternalLink> capacity are within acceptable values for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. If any of these metrics is above healthy limits, consider <InternalLink path="cockroach-start">adding nodes</InternalLink> to your cluster before beginning your upgrade.
2. Make sure your cluster is behind a <InternalLink path="recommended-production-settings#load-balancing">load balancer</InternalLink>, or your clients are configured to talk to multiple nodes. If your application communicates with only a single node, stopping that node to upgrade its CockroachDB binary will cause your application to fail.
3. By default, the <InternalLink path="architecture/storage-layer">storage engine</InternalLink> uses a compaction concurrency of 3. If you have sufficient IOPS and CPU headroom, you can consider increasing this setting via the `COCKROACH_COMPACTION_CONCURRENCY` environment variable. This may help to reshape the LSM more quickly in <InternalLink path="architecture/storage-layer#inverted-lsms">inverted LSM</InternalLink> scenarios; and it can lead to increased overall performance for some workloads. Cockroach Labs **strongly recommends** testing your workload against non-default values of this setting.
4. CockroachDB is designed with high fault tolerance. However, taking regular backups of your data is an operational best practice for <InternalLink path="disaster-recovery-planning">disaster recovery</InternalLink> planning. Refer to <InternalLink path="restoring-backups-across-versions#support-for-restoring-backups-into-a-newer-version">Restoring backups across versions</InternalLink>.
5. Review the <InternalLink version="releases" path="v26.1">v26.1 Release Notes</InternalLink>, as well as the release notes for any skipped major version. Pay careful attention to the sections for backward-incompatible changes, deprecations, changes to default cluster settings, and features that are not available until the upgrade is finalized.
6. Optionally [disable auto-finalization](#disable-auto-finalization) to preserve the ability to roll back a major-version upgrade instead of finalizing it. If auto-finalization is disabled, a major-version upgrade is not complete until it is finalized.

### Ensure you have a valid license key

To perform major version upgrades, you must have <InternalLink path="licensing-faqs#obtain-a-license">a valid license key</InternalLink>.

Patch version upgrades can be performed without a valid license key, with the following limitations:

* The cluster will run without limitations for a specified grace period. During that time, alerts are displayed that the cluster needs a valid license key. For more information, refer to the <InternalLink path="licensing-faqs">Licensing FAQs</InternalLink>.
* The cluster is <InternalLink path="licensing-faqs">throttled</InternalLink> at the end of the grace period if no <InternalLink path="licensing-faqs#set-a-license">valid license key is added to the cluster</InternalLink> before then.

If you have an **Enterprise Free** or **Enterprise Trial** license, you must enable telemetry using the <InternalLink path="cluster-settings">`diagnostics.reporting.enabled`</InternalLink> cluster setting, as shown below in order to finalize a major version upgrade:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING diagnostics.reporting.enabled = true;
```

If a cluster with an **Enterprise Free** or **Enterprise Trial** license is upgraded across patch versions and does not meet telemetry requirements:

* The cluster will run without limitations for a 7-day grace period. During that time, alerts are displayed that the cluster needs to send telemetry.
* The cluster is <InternalLink path="licensing-faqs">throttled</InternalLink> if telemetry is not received before the end of the grace period.

For more information, refer to the <InternalLink path="licensing-faqs">Licensing FAQs</InternalLink>.

If you want to stay on the previous version, you can roll back the upgrade before finalization.

## Perform a patch upgrade

1. Add a [partition](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set#staging-an-update) to the update strategy defined in the StatefulSet. Only the pods numbered greater than or equal to the partition value will be updated. For a cluster with 3 pods (e.g., `cockroachdb-0`, `cockroachdb-1`, `cockroachdb-2`) the partition value should be 2:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl patch statefulset cockroachdb \
   -p='{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   statefulset.apps/cockroachdb patched
   ```
2. Change the container image in the StatefulSet:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl patch statefulset cockroachdb \
   --type='json' \
   -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cockroachdb/cockroach:v24.3.29"}]'
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   statefulset.apps/cockroachdb patched
   ```
3. To check the status of the rolling upgrade, run `kubectl get pods`.
4. Verify that all pods have been upgraded:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl get pods \
   -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
   ```

   You can also check the CockroachDB version of each node in the <InternalLink path="ui-cluster-overview-page#node-details">DB Console</InternalLink>.

### Roll back a patch upgrade

To roll back a patch upgrade, repeat the steps in [Perform a patch upgrade](#perform-a-patch-upgrade), but configure the container image for the pods to the previous major version.

## Perform a major-version upgrade

To perform a major upgrade:

1. 1. Add a [partition](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set#staging-an-update) to the update strategy defined in the StatefulSet. Only the pods numbered greater than or equal to the partition value will be updated. For a cluster with 3 pods (e.g., `cockroachdb-0`, `cockroachdb-1`, `cockroachdb-2`) the partition value should be 2:

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ kubectl patch statefulset cockroachdb \
-p='{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'
```

```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
statefulset.apps/cockroachdb patched
```

1. Change the container image in the StatefulSet:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl patch statefulset cockroachdb \
   --type='json' \
   -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"cockroachdb/cockroach:v26.1.1"}]'
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   statefulset.apps/cockroachdb patched
   ```
2. To check the status of the rolling upgrade, run `kubectl get pods`.
3. After the pod has been restarted with the new image, start the CockroachDB <InternalLink path="cockroach-sql">built-in SQL client</InternalLink>:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl exec -it cockroachdb-client-secure \-- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=cockroachdb-public
   ```
4. Run the following SQL query to verify that the number of underreplicated ranges is zero:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SELECT sum((metrics->>'ranges.underreplicated')::DECIMAL)::INT AS ranges_underreplicated FROM crdb_internal.kv_store_status;
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
     ranges_underreplicated
   --------------------------
                          0
   (1 row)
   ```

   This indicates that it is safe to proceed to the next pod.
5. Exit the SQL shell:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > \q
   ```
6. Decrement the partition value by 1 to allow the next pod in the cluster to update:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl patch statefulset cockroachdb \
   -p='{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":1}}}}'
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   statefulset.apps/cockroachdb patched
   ```
7. Repeat steps 4-8 until all pods have been restarted and are running the new image (the final partition value should be `0`).
8. Check the image of each pod to confirm that all have been upgraded:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl get pods \
   -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   cockroachdb-0   cockroachdb/cockroach:v26.1.1
   cockroachdb-1   cockroachdb/cockroach:v26.1.1
   cockroachdb-2   cockroachdb/cockroach:v26.1.1
   ...
   ```

   You can also check the CockroachDB version of each node in the <InternalLink path="ui-cluster-overview-page#node-details">DB Console</InternalLink>.
9. If auto-finalization is disabled, the upgrade is not complete until you [finalize the upgrade](#finalize-a-major-version-upgrade-manually).

### Finalize a major-version upgrade manually

To finalize a major-version upgrade:

1. Connect to the cluster using the SQL shell:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=cockroachdb-public
   ```
2. Run the following command:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > RESET CLUSTER SETTING cluster.preserve_downgrade_option;
   ```

   A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return the previous version\`.

   You can monitor the process of the migration in the DB Console <InternalLink path="ui-jobs-page">Jobs page</InternalLink>. Migration jobs have names in the format `26.1-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs.

   The amount of time required for finalization depends on the amount of data in the cluster, because finalization runs various internal maintenance and migration tasks. During this time, the cluster will experience a small amount of additional load.

   When all migration jobs have completed, the upgrade is complete.
3. To confirm that finalization has completed, check the cluster version:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > SHOW CLUSTER SETTING version;
   ```

   If the cluster continues to report that it is on the previous version, finalization has not completed. If auto-finalization is enabled but finalization has not completed, check for the existence of <InternalLink path="node-shutdown?filters=decommission#status-change">decommissioning nodes</InternalLink> where decommission has stalled. In most cases, issuing the `decommission` command again resolves the issue. If you have trouble upgrading, [contact Support](https://cockroachlabs.com/support/hc).

### Roll back a major-version upgrade

To roll back to the previous major version before an upgrade is finalized:

1. Change the container image in the custom resource to use the previous major version:

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   image:
     name: cockroachdb/cockroach:v26.1
   ```
2. Apply the new settings to the cluster:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   kubectl apply -f example.yaml
   ```

   The Operator will perform the staged rollback.
3. To check the status of the rollback, run `kubectl get pods`.
4. Verify that all pods have been rolled back:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl get pods \
   -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
   ```

Rollbacks do not require finalization.

## Disable auto-finalization

By default, auto-finalization is enabled, and a major-version upgrade is finalized when all nodes have rejoined the cluster using the new `cockroach` binary. This means that by default, a major-version upgrade cannot be rolled back. Instead, you must <InternalLink path="restoring-backups-across-versions#support-for-restoring-backups-into-a-newer-version">restore the cluster to the previous version</InternalLink>.

To disable auto-finalization:

1. Connect to the cluster using the SQL shell:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=cockroachdb-public
   ```
2. Set the <InternalLink path="cluster-settings">cluster setting</InternalLink> `cluster.auto_upgrade.enabled` to `false`.

Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade-manually) or [roll it back](#roll-back-a-major-version-upgrade).

<Note>
  Previously, to disable automatic finalization and preserve the ability to roll back a major-version upgrade, it was required to set the cluster setting `cluster.preserve_downgrade_option` to the cluster's current major version before beginning the major-version upgrade, and then to unset the setting to finalize the upgrade.

  We now recommend managing a cluster's finalization policy using the cluster setting `cluster.auto_upgrade.enabled`, which was introduced in v23.2. The setting does not need to be modified after it is initially set.

  Either of these settings prevents automatic finalization.
</Note>

## Before you begin

If you <InternalLink path="deploy-cockroachdb-with-kubernetes-openshift">deployed CockroachDB on Red Hat OpenShift</InternalLink>, substitute `kubectl` with `oc` in the following commands.

All `kubectl` steps should be performed in the <InternalLink path="deploy-cockroachdb-with-kubernetes#install-the-operator">namespace where you installed the operator</InternalLink>. By default, this is `cockroach-operator-system`.

Before beginning a major-version or patch upgrade:

1. Verify the overall health of your cluster using the <InternalLink path="ui-cluster-overview-page">DB Console</InternalLink>:
   * Under **Node Status**, make sure all nodes that should be live are listed as such. If any nodes are unexpectedly listed as `SUSPECT` or `DEAD`, identify why the nodes are offline and either restart them or <InternalLink path="node-shutdown?filters=decommission#remove-nodes">decommission</InternalLink> them before beginning your upgrade. If there are `DEAD` and non-decommissioned nodes in your cluster, the upgrade cannot be finalized. If any node is not fully decommissioned, try the following:
     1. First, reissue the <InternalLink path="node-shutdown?filters=decommission#decommission-the-node">decommission command</InternalLink>. The second command typically succeeds within a few minutes.
     2. If the second decommission command does not succeed, <InternalLink path="node-shutdown?filters=decommission#recommission-nodes">recommission</InternalLink> and then decommission it again. Before continuing the upgrade, the node must be marked as `decommissioned`.
   * Under **Replication Status**, make sure there are `0` under-replicated and unavailable ranges. Otherwise, performing a rolling upgrade increases the risk that ranges will lose a majority of their replicas and cause cluster unavailability. Therefore, it's important to identify and resolve the cause of range under-replication and/or unavailability before beginning your upgrade.
   * In the **Node List**, make sure all nodes are on the same version. Upgrade them to the cluster's current version before continuing. If any nodes are behind, this also indicates that the previous major-version upgrade may not be finalized.
   * In the **Metrics** dashboards, make sure <InternalLink path="common-issues-to-monitor#cpu-usage">CPU</InternalLink>, <InternalLink path="common-issues-to-monitor#database-memory-usage">memory</InternalLink>, and <InternalLink path="common-issues-to-monitor#storage-capacity">storage</InternalLink> capacity are within acceptable values for each node. Nodes must be able to tolerate some increase in case the new version uses more resources for your workload. If any of these metrics is above healthy limits, consider <InternalLink path="cockroach-start">adding nodes</InternalLink> to your cluster before beginning your upgrade.
2. Make sure your cluster is behind a <InternalLink path="recommended-production-settings#load-balancing">load balancer</InternalLink>, or your clients are configured to talk to multiple nodes. If your application communicates with only a single node, stopping that node to upgrade its CockroachDB binary will cause your application to fail.
3. By default, the <InternalLink path="architecture/storage-layer">storage engine</InternalLink> uses a compaction concurrency of 3. If you have sufficient IOPS and CPU headroom, you can consider increasing this setting via the `COCKROACH_COMPACTION_CONCURRENCY` environment variable. This may help to reshape the LSM more quickly in <InternalLink path="architecture/storage-layer#inverted-lsms">inverted LSM</InternalLink> scenarios; and it can lead to increased overall performance for some workloads. Cockroach Labs **strongly recommends** testing your workload against non-default values of this setting.
4. CockroachDB is designed with high fault tolerance. However, taking regular backups of your data is an operational best practice for <InternalLink path="disaster-recovery-planning">disaster recovery</InternalLink> planning. Refer to <InternalLink path="restoring-backups-across-versions#support-for-restoring-backups-into-a-newer-version">Restoring backups across versions</InternalLink>.
5. Review the <InternalLink version="releases" path="v26.1">v26.1 Release Notes</InternalLink>, as well as the release notes for any skipped major version. Pay careful attention to the sections for backward-incompatible changes, deprecations, changes to default cluster settings, and features that are not available until the upgrade is finalized.
6. Optionally [disable auto-finalization](#disable-auto-finalization) to preserve the ability to roll back a major-version upgrade instead of finalizing it. If auto-finalization is disabled, a major-version upgrade is not complete until it is finalized.

### Ensure you have a valid license key

To perform major version upgrades, you must have <InternalLink path="licensing-faqs#obtain-a-license">a valid license key</InternalLink>.

Patch version upgrades can be performed without a valid license key, with the following limitations:

* The cluster will run without limitations for a specified grace period. During that time, alerts are displayed that the cluster needs a valid license key. For more information, refer to the <InternalLink path="licensing-faqs">Licensing FAQs</InternalLink>.
* The cluster is <InternalLink path="licensing-faqs">throttled</InternalLink> at the end of the grace period if no <InternalLink path="licensing-faqs#set-a-license">valid license key is added to the cluster</InternalLink> before then.

If you have an **Enterprise Free** or **Enterprise Trial** license, you must enable telemetry using the <InternalLink path="cluster-settings">`diagnostics.reporting.enabled`</InternalLink> cluster setting, as shown below in order to finalize a major version upgrade:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
SET CLUSTER SETTING diagnostics.reporting.enabled = true;
```

If a cluster with an **Enterprise Free** or **Enterprise Trial** license is upgraded across patch versions and does not meet telemetry requirements:

* The cluster will run without limitations for a 7-day grace period. During that time, alerts are displayed that the cluster needs to send telemetry.
* The cluster is <InternalLink path="licensing-faqs">throttled</InternalLink> if telemetry is not received before the end of the grace period.

For more information, refer to the <InternalLink path="licensing-faqs">Licensing FAQs</InternalLink>.

If you want to stay on the previous version, you can roll back the upgrade before finalization.

## Perform a patch upgrade

1. Add a [partition](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set#staging-an-update) to the update strategy defined in the StatefulSet. Only the pods numbered greater than or equal to the partition value will be updated. For a cluster with 3 pods (e.g., `cockroachdb-0`, `cockroachdb-1`, `cockroachdb-2`) the partition value should be 2:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ helm upgrade \
   my-release \
   cockroachdb/cockroachdb \
   --set statefulset.updateStrategy.rollingUpdate.partition=2
   ```
2. Connect to the cluster using the SQL shell:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=my-release-cockroachdb-public
   ```
3. Remove the cluster initialization job from when the cluster was created:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl delete job my-release-cockroachdb-init
   ```
4. Change the container image in the StatefulSet:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ helm upgrade \
   my-release \
   cockroachdb/cockroachdb \
   --set image.tag=v24.3.29 \
   --reuse-values
   ```
5. To check the status of the rolling upgrade, run `kubectl get pods`.
6. Verify that all pods have been upgraded:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl get pods \
   -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
   ```

   You can also check the CockroachDB version of each node in the <InternalLink path="ui-cluster-overview-page#node-details">DB Console</InternalLink>.

### Roll back a patch upgrade

To roll back a patch upgrade, repeat the steps in [Perform a patch upgrade](#perform-a-patch-upgrade), but configure the container image for the pods to the previous major version.

## Perform a major-version upgrade

To perform a major upgrade:

1. Add a [partition](https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set#staging-an-update) to the update strategy defined in the StatefulSet. Only the pods numbered greater than or equal to the partition value will be updated. For a cluster with 3 pods (e.g., `cockroachdb-0`, `cockroachdb-1`, `cockroachdb-2`) the partition value should be 2:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ helm upgrade \
   my-release \
   cockroachdb/cockroachdb \
   --set statefulset.updateStrategy.rollingUpdate.partition=2
   ```
2. Connect to the cluster using the SQL shell:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=my-release-cockroachdb-public
   ```
3. Remove the cluster initialization job from when the cluster was created:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl delete job my-release-cockroachdb-init
   ```
4. Change the container image in the StatefulSet:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ helm upgrade \
   my-release \
   cockroachdb/cockroachdb \
   --set image.tag=v26.1.1 \
   --reuse-values
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   NAME                                READY     STATUS              RESTARTS   AGE
   my-release-cockroachdb-0            1/1       Running             0          2m
   my-release-cockroachdb-1            1/1       Running             0          3m
   my-release-cockroachdb-2            0/1       ContainerCreating   0          25s
   my-release-cockroachdb-init-nwjkh   0/1       ContainerCreating   0          6s
   ...
   ```

<Note>
  Ignore the pod for cluster initialization. It is re-created as a byproduct of the StatefulSet configuration but does not impact your existing cluster.
</Note>

5. After the pod has been restarted with the new image, start the CockroachDB <InternalLink path="cockroach-sql">built-in SQL client</InternalLink>:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=my-release-cockroachdb-public
   ```
6. Run the following SQL query to verify that the number of underreplicated ranges is zero:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   SELECT sum((metrics->>'ranges.underreplicated')::DECIMAL)::INT AS ranges_underreplicated FROM crdb_internal.kv_store_status;
   ```

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
     ranges_underreplicated
   --------------------------
                          0
   (1 row)
   ```

   This indicates that it is safe to proceed to the next pod.
7. Exit the SQL shell:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > \q
   ```
8. Decrement the partition value by 1 to allow the next pod in the cluster to update:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ helm upgrade \
   my-release \
   cockroachdb/cockroachdb \
   --set statefulset.updateStrategy.rollingUpdate.partition=1 \
   ```
9. Repeat steps 4-8 until all pods have been restarted and are running the new image (the final partition value should be `0`).
10. Check the image of each pod to confirm that all have been upgraded:

    ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
    $ kubectl get pods \
    -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
    ```

    ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
    my-release-cockroachdb-0    cockroachdb/cockroach:v26.1.1
    my-release-cockroachdb-1    cockroachdb/cockroach:v26.1.1
    my-release-cockroachdb-2    cockroachdb/cockroach:v26.1.1
    ...
    ```

    You can also check the CockroachDB version of each node in the <InternalLink path="ui-cluster-overview-page#node-details">DB Console</InternalLink>.
11. If auto-finalization is disabled, the upgrade is not complete until you [finalize the upgrade](#finalize-a-major-version-upgrade-manually).

### Finalize a major-version upgrade manually

To finalize a major-version upgrade:

1. Connect to the cluster using the SQL shell:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=cockroachdb-public
   ```
2. Run the following command:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > RESET CLUSTER SETTING cluster.preserve_downgrade_option;
   ```

   A series of migration jobs runs to enable certain types of features and changes in the new major version that cannot be rolled back. These include changes to system schemas, indexes, and descriptors, and enabling certain types of improvements and new features. Until the upgrade is finalized, these features and functions will not be available and the command `SHOW CLUSTER SETTING version` will return the previous version\`.

   You can monitor the process of the migration in the DB Console <InternalLink path="ui-jobs-page">Jobs page</InternalLink>. Migration jobs have names in the format `26.1-{migration-id}`. If a migration job fails or stalls, Cockroach Labs can use the migration ID to help diagnose and troubleshoot the problem. Each major version has different migration jobs with different IDs.

   The amount of time required for finalization depends on the amount of data in the cluster, because finalization runs various internal maintenance and migration tasks. During this time, the cluster will experience a small amount of additional load.

   When all migration jobs have completed, the upgrade is complete.
3. To confirm that finalization has completed, check the cluster version:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > SHOW CLUSTER SETTING version;
   ```

   If the cluster continues to report that it is on the previous version, finalization has not completed. If auto-finalization is enabled but finalization has not completed, check for the existence of <InternalLink path="node-shutdown?filters=decommission#status-change">decommissioning nodes</InternalLink> where decommission has stalled. In most cases, issuing the `decommission` command again resolves the issue. If you have trouble upgrading, [contact Support](https://cockroachlabs.com/support/hc).

### Roll back a major-version upgrade

To roll back to the previous major version before an upgrade is finalized:

1. Change the container image in the custom resource to use the previous major version:

   ```text theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   image:
     name: cockroachdb/cockroach:v26.1
   ```
2. Apply the new settings to the cluster:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   kubectl apply -f example.yaml
   ```

   The Operator will perform the staged rollback.
3. To check the status of the rollback, run `kubectl get pods`.
4. Verify that all pods have been rolled back:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl get pods \
   -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}'
   ```

Rollbacks do not require finalization.

## Disable auto-finalization

By default, auto-finalization is enabled, and a major-version upgrade is finalized when all nodes have rejoined the cluster using the new `cockroach` binary. This means that by default, a major-version upgrade cannot be rolled back. Instead, you must <InternalLink path="restoring-backups-across-versions#support-for-restoring-backups-into-a-newer-version">restore the cluster to the previous version</InternalLink>.

To disable auto-finalization:

1. Connect to the cluster using the SQL shell:

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   $ kubectl exec -it cockroachdb-client-secure \
   -- ./cockroach sql \
   --certs-dir=/cockroach-certs \
   --host=cockroachdb-public
   ```
2. Set the <InternalLink path="cluster-settings">cluster setting</InternalLink> `cluster.preserve_downgrade_option` to the cluster's current major version. The Helm chart does not yet support the <InternalLink path="cluster-settings">cluster setting</InternalLink> `cluster.auto_upgrade.enabled`.

Now, to complete a major-version upgrade, you must manually [finalize it](#finalize-a-major-version-upgrade-manually) or [roll it back](#roll-back-a-major-version-upgrade).

## Troubleshooting

After the upgrade has finalized (whether manually or automatically), it is no longer possible to roll back the upgrade. If you are experiencing problems, we recommend that you <InternalLink path="support-resources">open a support request</InternalLink> for assistance.

In the event of catastrophic failure or corruption, it may be necessary to <InternalLink path="restore">restore</InternalLink> from a backup to a new cluster running the previous major version.

## See also