Physical Cluster Replication Technical Overview

continuously and asynchronously replicates data from an active primary CockroachDB cluster to a passive standby cluster. When both clusters are virtualized, each cluster contains a system virtual cluster and an application during the PCR stream:

• The system virtual cluster manages the cluster’s control plane and the replication of the cluster’s data. Admins connect to the system virtual cluster to configure and manage the underlying CockroachDB cluster, set up PCR, create and manage a virtual cluster, and observe metrics and logs for the CockroachDB cluster and each virtual cluster.
• The application virtual cluster manages the cluster’s data plane. Application virtual clusters contain user data and run application workloads.

If you utilize the read on standby feature in PCR, the standby cluster has an additional reader virtual cluster that safely serves read requests on the replicating virtual cluster.

Replication stream start-up sequence

stream consists of two jobs: one each on the standby and primary cluster:

• Standby consumer job: Communicates with the primary cluster via an ordinary SQL connection and is responsible for initiating the replication stream. The consumer job ingests updates from the primary cluster producer job.
• Primary producer job: Protects data on the primary cluster and sends updates to the standby cluster.

The stream initialization proceeds as follows:

1. The standby’s consumer job connects to the primary cluster via the standby’s system virtual cluster and starts the primary cluster’s REPLICATION STREAM PRODUCER job.
2. The primary cluster chooses a timestamp at which to start the physical replication stream. Data on the primary is protected from until it is replicated to the standby using a .
3. The primary cluster returns the timestamp and a for the replication job.
4. The standby cluster retrieves a list of all nodes in the primary cluster. It uses this list to distribute work across all nodes in the standby cluster.
5. The initial scan runs on the primary and backfills all data from the primary virtual cluster as of the starting timestamp of the replication stream.
6. Once the initial scan is complete, the primary then begins streaming all changes from the point of the starting timestamp.

During the replication stream

The replication happens at the byte level, which means that the job is unaware of databases, tables, row boundaries, and so on. However, when a failover to the standby cluster is initiated, the replication job ensures that the cluster is in a transactionally consistent state as of a certain point in time. Beyond the application data, the job will also replicate users, privileges, basic zone configuration, and schema changes.

During the job, are periodically emitting resolved timestamps, which is the time where the ingested data is known to be consistent. Resolved timestamps provide a guarantee that there are no new writes from before that timestamp. This allows the standby cluster to move the forward as the replicated timestamp advances. This information is sent to the primary cluster, which allows for to continue as the replication stream on the standby cluster advances.

If the primary cluster does not receive replicated time information from the standby after 24 hours, it cancels the replication job. This ensures that an inactive replication job will not prevent garbage collection.

Failover and promotion process

The tracked replicated time and the advancing protected timestamp allow the replication stream to also track retained time, which is a timestamp in the past indicating the lower bound that the replication stream could fail over to. The retained time can be up to 4 hours in the past, due to the protected timestamp. Therefore, the failover window for a replication job falls between the retained time and the replicated time.

Replication lag is the time between the most up-to-date replicated time and the actual time. While the replication keeps as current as possible to the actual time, this replication lag window is where there is potential for data loss.

For the , the standby cluster waits until it has reached the specified failover time, which can be in the (retained time), the timestamp, or in the . Once that timestamp has been reached, the replication stream stops and any data in the standby cluster that is above the failover time is removed. Depending on how much data the standby needs to revert, this can affect the duration of RTO (recovery time objective).

After reverting any necessary data, the standby virtual cluster is promoted as available to serve traffic and the replication job ends.

For detail on failing back to the primary cluster following a failover, refer to .