> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cockroachlabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Disaster Recovery Planning

CockroachDB is built to be [fault-tolerant and to recover automatically](https://docs.cockroachlabs.com/docs/stable/demo-cockroachdb-resilience.md), but sometimes disasters happen. A *disaster* is any event that puts your cluster at risk, and usually means your cluster is experiencing [hardware failure](#hardware-failure), [data failure](#data-failure), or has [compromised security keys](#compromised-security-keys). Having a disaster recovery plan enables you to recover quickly, while limiting the consequences.

## Hardware failure

When planning to survive hardware failures, start by determining the minimum replication factor you need based on your fault tolerance goals:

* [Single-region survivability planning](#single-region-survivability-planning)
* [Multi-region survivability planning](#multi-region-survivability-planning)

> [!CAUTION]
>
> Increasing the replication factor can impact write performance in that more replicas must agree to reach quorum. For more details about the mechanics of writes and the Raft protocol, see [Read and Writes Overview](https://docs.cockroachlabs.com/docs/stable/architecture/reads-and-writes-overview.md).

> [!NOTE]
>
> For the purposes of choosing a replication factor, disk failure is equivalent to node failure.

If you experience hardware failures in a cluster, the recovery actions you need to take will depend on the type of infrastructure and topology pattern used:

* [Single-region recovery](#single-region-recovery)
* [Multi-region recovery](#multi-region-recovery)

### Single-region survivability planning

The table below shows the replication factor (RF) needed to achieve the listed fault tolerance goals for a single region, cloud-deployed cluster with nodes spread as evenly as possible across 3 availability zones (AZs):

> [!NOTE]
>
> See our [basic production topology](https://docs.cockroachlabs.com/docs/stable/topology-basic-production.md#configuration) for configuration guidance.

| Fault Tolerance Goals | 3 nodes | 5 nodes | 9 nodes |
| - | - | - | - |
| 1 Node | RF = 3 | RF = 3 | RF = 3 |
| 1 AZ | RF = 3 | RF = 3 | RF = 3 |
| 2 Nodes | Not possible | RF = 5 | RF = 5 |
| AZ + Node | Not possible | Not possible | RF = 9 |
| 2 AZ | Not possible | Not possible | Not possible |

To be able to survive 2+ availability zones failing, scale to a [multi-region](#multi-region-survivability-planning) deployment.

### Single-region recovery

For hardware failures in a single-region cluster, the recovery actions vary and depend on the type of infrastructure used.

For example, consider a cloud-deployed CockroachDB cluster with the following setup:

* Single-region
* 3 nodes
* A node in each availability zone (i.e., 3 AZs)
* Replication factor of 3

The table below describes what actions to take to recover from various hardware failures in this example cluster:

| Failure | Availability | Consequence | Action to Take |
| - | - | - | - |
| 1 Disk | √ | Fewer resources are available. Some data will be under-replicated until the failed nodes are marked dead. Once marked dead, data is replicated to other nodes and the cluster remains healthy. | [Restart the node](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md) with a new disk. |
| 1 Node | √ | If the node or AZ becomes unavailable, check the [Overview dashboard](https://docs.cockroachlabs.com/docs/stable/ui-overview-dashboard.md) on the DB Console: If the down node is marked Suspect, try [restarting the node](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md). If the down node is marked Dead, [decommission the node](https://docs.cockroachlabs.com/docs/stable/node-shutdown.md?filters=decommission#remove-a-dead-node), wipe the store path, and then [rejoin it back to the cluster](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md). If the node has additional hardware issues, decommission the node and [add a new node](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md) to the cluster. Ensure that [locality flags are set](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md#locality) correctly upon node startup. | |
| 1 AZ | √ | | |
| 2 Nodes | X | Cluster is unavailable. | [Restart](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md) 1 of the 2 nodes that are down to regain quorum. If you can’t recover at least 1 node, [contact Cockroach Labs support](https://support.cockroachlabs.com) for assistance. |
| 1 AZ + 1 Node | X | Cluster is unavailable. | [Restart](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md) the node that is down to regain quorum. When the AZ comes back online, try restarting the node. If you can’t recover at least 1 node, [contact Cockroach Labs support](https://support.cockroachlabs.com) for assistance. |
| 2 AZ | X | Cluster is unavailable. | When the AZ comes back online, try [restarting](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md) at least 1 of the nodes. You can also [contact Cockroach Labs support](https://support.cockroachlabs.com) for assistance. |
| 3 Nodes | X | Cluster is unavailable. | [Restart](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md) 2 of the 3 nodes that are down to regain quorum. If you can’t recover 2 of the 3 failed nodes, [contact Cockroach Labs support](https://support.cockroachlabs.com) for assistance. |
| 1 Region | X | Cluster is unavailable. Potential data loss between last backup and time of outage if the region and nodes did not come back online. | When the region comes back online, try [restarting the nodes](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md) in the cluster. If the region does not come back online and nodes are lost or destroyed, try [restoring the latest cluster backup](https://docs.cockroachlabs.com/docs/stable/restore.md) into a new cluster. You can also [contact Cockroach Labs support](https://support.cockroachlabs.com) for assistance. |

> [!NOTE]
>
> When using Kubernetes, recovery actions happen automatically in many cases and no action needs to be taken.

### Multi-region survivability planning

> [!TIP]
>
> By default, every [multi-region database](https://docs.cockroachlabs.com/docs/stable/multiregion-overview.md) has a [zone-level survival goal](https://docs.cockroachlabs.com/docs/stable/multiregion-overview.md#survival-goals) associated with it.  The survival goal setting provides an abstraction that handles the low-level details of replica placement to ensure your desired fault tolerance.  The information below is still useful for legacy deployments.

The table below shows the replication factor (RF) needed to achieve the listed fault tolerance (e.g., survive 1 failed node) for a multi-region, cloud-deployed cluster with 3 availability zones (AZ) per region and one node in each AZ:

> [!CAUTION]
>
> The chart below describes the CockroachDB default behavior when locality flags are correctly set. It does not use geo-partitioning or a specific [topology pattern](https://docs.cockroachlabs.com/docs/stable/topology-patterns.md). For a multi-region cluster in production, we do not recommend using the default behavior, as the cluster's performance will be negatively affected.

| Fault Tolerance Goals | 3 Regions (9 Nodes Total) | 4 Regions (12 Nodes Total) | 5 Regions (15 Nodes Total) |
| - | - | - | - |
| 1 Node | RF = 3 | RF = 3 | RF = 3 |
| 1 AZ | RF = 3 | RF = 3 | RF = 3 |
| 1 Region | RF = 3 | RF = 3 | RF = 3 |
| 2 Nodes | RF = 5 | RF = 5 | RF = 5 |
| 1 Region + 1 Node | RF = 9 | RF = 7 | RF = 5 |
| 2 Regions | Not possible | Not possible | RF = 5 |
| 2 Regions + 1 Node | Not possible | Not possible | RF = 15 |

### Multi-region recovery

For hardware failures in a multi-region cluster, the actions taken to recover vary and depend on the type of infrastructure used.

For example, consider a cloud-deployed CockroachDB cluster with the following setup:

* 3 regions
* 3 AZs per region
* 9 nodes (1 node per AZ)
* Replication factor of 3

The table below describes what actions to take to recover from various hardware failures in this example cluster:

| Failure | Availability | Consequence | Action to Take |
| - | - | - | - |
| 1 Disk | √ | Under-replicated data. Fewer resources for workload. | [Restart the node](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md) with a new disk. |
| 1 AZ | √ | | |
| 1 Region | √ | Check the [Overview dashboard](https://docs.cockroachlabs.com/docs/stable/ui-overview-dashboard.md) on the DB Console. If nodes are marked Dead, [decommission the nodes](https://docs.cockroachlabs.com/docs/stable/node-shutdown.md?filters=decommission#remove-a-dead-node) and [add 3 new nodes](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md) in a new region. Ensure that [locality flags are set](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md#locality) correctly upon node startup. | |
| 2 or More Regions | X | Cluster is unavailable. Potential data loss between last backup and time of outage if the region and nodes did not come back online. | When the regions come back online, try [restarting the nodes](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md) in the cluster. If the regions do not come back online and nodes are lost or destroyed, try [restoring the latest cluster backup](https://docs.cockroachlabs.com/docs/stable/restore.md) into a new cluster. You can also [contact Cockroach Labs support](https://support.cockroachlabs.com) for assistance. |

When using Kubernetes, recovery actions happen automatically in many cases and no action needs to be taken.

## Data failure

When dealing with data failure due to bad actors, rogue applications, or data corruption, domain expertise is required to identify the affected rows and determine how to remedy the situation (e.g., remove the incorrectly inserted rows, insert deleted rows, etc.). However, there are a few actions that you can take for short-term remediation:

* If you are within the garbage collection window, [run differentials](#run-differentials).
* If you have a backup file, [restore to a point in time](#restore-to-a-point-in-time).
* If your cluster is running and you do not have a backup with the data you need, [create a new backup](#create-a-new-backup).
* To [recover from corrupted data in a database or table](#recover-from-corrupted-data-in-a-database-or-table), restore the corrupted object.

> [!TIP]
>
> To give yourself more time to recover and clean up the corrupted data, put your application in “read only” mode and only run [`AS OF SYSTEM TIME`](https://docs.cockroachlabs.com/docs/stable/as-of-system-time) queries from the application.

### Run differentials

If you are within the [garbage collection window](https://docs.cockroachlabs.com/docs/stable/configure-replication-zones.md#gc-ttlseconds), run [`AS OF SYSTEM TIME`](https://docs.cockroachlabs.com/docs/stable/as-of-system-time) queries and use [`CREATE TABLE AS … SELECT * FROM`](https://docs.cockroachlabs.com/docs/stable/create-table-as) to create comparison data and run differentials to find the offending rows to fix.

Alternatively, you can use [`CREATE TABLE AS ... AS OF SYSTEM TIME`](https://docs.cockroachlabs.com/docs/stable/create-table-as#populate-create-table-as-with-historical-data-using-as-of-system-time) to generate a copy of the data as of a specific [timestamp](https://docs.cockroachlabs.com/docs/stable/timestamp.md). This also requires being within the garbage collection window.

If you are outside of the garbage collection window, you will need to use a [backup](https://docs.cockroachlabs.com/docs/stable/backup.md#with-revision-history) to run comparisons.

### Restore to a point in time

* Use your [backup](https://docs.cockroachlabs.com/docs/stable/backup.md#with-revision-history) file to [restore to a point in time](https://docs.cockroachlabs.com/docs/stable/take-backups-with-revision-history-and-restore-from-a-point-in-time.md) where you are certain there was no corruption. Note that the backup must have been taken with [revision history](https://docs.cockroachlabs.com/docs/stable/backup.md#with-revision-history).

### Create a new backup

If your cluster is running, you do not have a backup that encapsulates the time you want to [restore](https://docs.cockroachlabs.com/docs/stable/restore.md) to, and the data you want to recover is still in the [garbage collection window](https://docs.cockroachlabs.com/docs/stable/configure-replication-zones.md#gc-ttlseconds), there are two actions you can take:

* Trigger a new [backup `with_revision_history`](https://docs.cockroachlabs.com/docs/stable/take-backups-with-revision-history-and-restore-from-a-point-in-time) and you will have a backup you can use to restore to the desired point in time within the [garbage collection window](https://docs.cockroachlabs.com/docs/stable/configure-replication-zones.md#gc-ttlseconds).

### Recover from corrupted data in a database or table

If you have corrupted data in a database or table, [restore](https://docs.cockroachlabs.com/docs/stable/restore.md) the object from a prior [backup](https://docs.cockroachlabs.com/docs/stable/backup.md#with-revision-history). If revision history is in the backup, you can restore from a [point in time](https://docs.cockroachlabs.com/docs/stable/take-backups-with-revision-history-and-restore-from-a-point-in-time.md).

Instead of dropping the corrupted table or database, we recommend [renaming the table](https://docs.cockroachlabs.com/docs/stable/alter-table.md#rename-to) or [renaming the database](https://docs.cockroachlabs.com/docs/stable/alter-database.md#rename-to) so you have historical data to compare to later. If you drop a database, the database cannot be referenced with `AS OF SYSTEM TIME` queries, and you will need to take a backup that is backdated to the system time when the database still existed.

> [!NOTE]
>
> If the table you are restoring has foreign keys, [careful consideration](https://docs.cockroachlabs.com/docs/stable/restore.md#remove-the-foreign-key-before-restore) should be applied to make sure data integrity is maintained during the restore process.

## Compromised security keys

CockroachDB maintains a secure environment for your data. However, there are bad actors who may find ways to gain access or expose important security information. In the event that this happens, there are a few things you can do to get ahead of a security issue:

* If you have [changefeeds to cloud storage sinks](#changefeeds-to-cloud-storage), cancel the changefeed job and restart it with new access credentials.
* If you are using [ Encryption At Rest](#encryption-at-rest), rotate the store key(s).
* If you are using [wire encryption in transit / TLS](#wire-encryption-/-tls), rotate your keys.

### Changefeeds to cloud storage

1. [Cancel the changefeed job](https://docs.cockroachlabs.com/docs/stable/cancel-job.md) immediately and [record the high water timestamp](https://docs.cockroachlabs.com/docs/stable/monitor-and-debug-changefeeds.md#monitor-a-changefeed) for where the changefeed was stopped.
2. Remove the access keys from the identity management system of your cloud provider and replace with a new set of access keys.
3. [Create a new changefeed](https://docs.cockroachlabs.com/docs/stable/create-changefeed.md#start-a-new-changefeed-where-another-ended) with the new access credentials using the last high water timestamp.

### Encryption at rest

If you believe the user-defined store keys have been compromised, quickly attempt to rotate your store keys that are being used for your encryption at rest setup. If this key has already been compromised and the store keys were rotated by a bad actor, the cluster should be wiped if possible and [restored](https://docs.cockroachlabs.com/docs/stable/restore.md) from a prior backup.

If the compromised keys were not rotated by a bad actor, quickly attempt to [rotate the store key](https://docs.cockroachlabs.com/docs/stable/security-reference/encryption.md#rotating-keys) by restarting each of the nodes with the old key and the new key. For an example on how to do this, see [Encryption](https://docs.cockroachlabs.com/docs/stable/encryption.md#changing-encryption-algorithm-or-keys).

Once all of the nodes are restarted with the new key, put in a request to revoke the old key from the Certificate Authority.

> [!NOTE]
>
> CockroachDB does not allow prior store keys to be used again.

### Wire Encryption / TLS

As a best practice, [keys should be rotated](https://docs.cockroachlabs.com/docs/stable/rotate-certificates.md). In the event that keys have been compromised, quickly attempt to rotate your keys. This can include rotating node certificates, client certificates, and the CA certificate.

## See also

* [Fault Tolerance & Recovery](https://docs.cockroachlabs.com/docs/stable/demo-cockroachdb-resilience.md)
* [Back up and Restore Data](https://docs.cockroachlabs.com/docs/stable/take-full-and-incremental-backups.md)
* [Topology Patterns](https://docs.cockroachlabs.com/docs/stable/topology-patterns.md)
* [Production Checklist](https://docs.cockroachlabs.com/docs/stable/recommended-production-settings.md)