> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cockroachlabs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Follower Reads Topology

In a multi-region deployment, [follower reads](https://docs.cockroachlabs.com/docs/stable/follower-reads.md) are a good choice for tables with the following requirements:

* Read latency must be low, but write latency can be higher.
* Reads can be historical.
* Rows in the table, and all latency-sensitive queries, **cannot** be tied to specific geographies (e.g., a reference table).
* Table data must remain available during a region failure.

If reads can use stale data, use [stale follower reads](https://docs.cockroachlabs.com/docs/stable/follower-reads.md#stale-follower-reads). If reads must be exactly up-to-date, use [global tables](https://docs.cockroachlabs.com/docs/stable/global-tables.md) to achieve [strong follower reads](https://docs.cockroachlabs.com/docs/stable/follower-reads.md#follower-read-types). Up-to-date reads are required by tables referenced by [foreign keys](https://docs.cockroachlabs.com/docs/stable/foreign-key.md), for example.

## Before you begin

### Fundamentals

* Multi-region topology patterns are almost always table-specific. If you haven't already, [review the full range of patterns](https://docs.cockroachlabs.com/docs/stable/topology-patterns.md#multi-region) to ensure you choose the right one for each of your tables.
* Review how data is replicated and distributed across a cluster, and how this affects performance. It is especially important to understand the concept of the "leaseholder". For a summary, see [Reads and Writes in CockroachDB](https://docs.cockroachlabs.com/docs/stable/architecture/reads-and-writes-overview.md). For a deeper dive, see the CockroachDB [Architecture Overview](https://docs.cockroachlabs.com/docs/stable/architecture/overview.md).
* Review the concept of [locality](https://docs.cockroachlabs.com/docs/stable/cockroach-start.md#locality), which CockroachDB uses to place and balance data based on how you define [replication controls](https://docs.cockroachlabs.com/docs/stable/configure-replication-zones.md).
* Review the recommendations and requirements in our [Production Checklist](https://docs.cockroachlabs.com/docs/stable/recommended-production-settings.md).
* This topology doesn't account for hardware specifications, so be sure to follow our [hardware recommendations](https://docs.cockroachlabs.com/docs/stable/recommended-production-settings.md#hardware) and perform a POC to size hardware for your use case. For optimal cluster performance, Cockroach Labs recommends that all nodes use the same hardware and operating system.
* Adopt relevant [SQL Best Practices](https://docs.cockroachlabs.com/docs/stable/performance-best-practices-overview.md) to ensure optimal performance.

### Cluster setup

Each [multi-region pattern](https://docs.cockroachlabs.com/docs/stable/topology-patterns.md#multi-region) assumes the following setup:

![Multi-region hardware setup](https://mintcdn.com/cockroachlabs/iiBVxthghoAOO3UM/images/common/topology-patterns/topology_multi-region_hardware.png?fit=max\&auto=format\&n=iiBVxthghoAOO3UM\&q=85\&s=810cffba5f7294be67bc34e0b3276c28)

#### Hardware

* 3 regions
* Per region, 3+ AZs with 3+ VMs evenly distributed across them
* Region-specific app instances and load balancers
  * Each load balancer redirects to CockroachDB nodes in its region.
  * When CockroachDB nodes are unavailable in a region, the load balancer redirects to nodes in other regions.

#### Cluster startup

Start each node with the [`--locality`](https://docs.cockroachlabs.com/docs/stable/cockroach-start#locality) flag specifying its region and AZ combination. For example, the following command starts a node in the `west1` AZ of the `us-west` region:

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
$ cockroach start \
--locality=region=us-west,zone=west1 \
--certs-dir=certs \
--advertise-addr=<node1 internal address \
--join=<node1 internal address:26257,<node2 internal address:26257,<node3 internal address:26257 \
--cache=.25 \
--max-sql-memory=.25 \
--background
```

## Configuration

With each node started with the [`--locality`](https://docs.cockroachlabs.com/docs/stable/cockroach-start#locality) flag specifying its region and zone combination, CockroachDB will balance the replicas for a table across the three regions:

![Follower reads table replication](https://mintcdn.com/cockroachlabs/iiBVxthghoAOO3UM/images/common/topology-patterns/topology_follower_reads1.png?fit=max\&auto=format\&n=iiBVxthghoAOO3UM\&q=85\&s=e3403abd96ae2eca606ab48bb98b5223)

### Summary

You configure your application to use [follower reads](https://docs.cockroachlabs.com/docs/stable/follower-reads.md) by adding an [`AS OF SYSTEM TIME`](https://docs.cockroachlabs.com/docs/stable/as-of-system-time) clause when reading from the table. With this clause CockroachDB reads slightly historical data from the closest replica so as to avoid being routed to the leaseholder, which may be in an entirely different region. Writes, however, will still leave the region to get consensus for the table.

### Steps

1. Create the `postal_codes` table:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > CREATE TABLE postal_codes (
       id INT PRIMARY KEY,
       code STRING
   );
   ```

2. Insert data into the `postal_codes` table:

   ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   > INSERT INTO postal_codes (ID, code) VALUES (1, '10001'), (2, '10002'), (3, '10003'), (4,'60601'), (5,'60602'), (6,'60603'), (7,'90001'), (8,'90002'), (9,'90003');
   ```

3. Decide which type of follower read to perform: exact staleness or  bounded staleness. For more information about when to use each type of read, see [when to use exact staleness reads](https://docs.cockroachlabs.com/docs/stable/follower-reads.md#when-to-use-exact-staleness-reads) and [when to use bounded staleness reads](https://docs.cockroachlabs.com/docs/stable/follower-reads.md#when-to-use-bounded-staleness-reads).
   * To use [exact staleness follower reads](https://docs.cockroachlabs.com/docs/stable/follower-reads.md#exact-staleness-reads), configure your app to use [`AS OF SYSTEM TIME`](https://docs.cockroachlabs.com/docs/stable/as-of-system-time) with the [`follower_read_timestamp()` function](https://docs.cockroachlabs.com/docs/stable/functions-and-operators#date-and-time-functions) whenever reading from the table:

     ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
     > SELECT code FROM postal_codes
         AS OF SYSTEM TIME follower_read_timestamp()
                 WHERE id = 5;
     ```

     You can also set the `AS OF SYSTEM TIME` value for all operations in a read-only transaction:

     ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
     > BEGIN;

     SET TRANSACTION AS OF SYSTEM TIME follower_read_timestamp();

       SELECT code FROM postal_codes
         WHERE id = 5;

       SELECT code FROM postal_codes
         WHERE id = 6;

     COMMIT;
     ```

> [!TIP]
>
> Using the [`SET TRANSACTION`](https://docs.cockroachlabs.com/docs/stable/set-transaction#use-the-as-of-system-time-option) statement as shown in the preceding example will make it easier to use exact staleness follower reads from [drivers and ORMs](https://docs.cockroachlabs.com/docs/stable/install-client-drivers.md).

* To use [bounded staleness follower reads](https://docs.cockroachlabs.com/docs/stable/follower-reads.md#bounded-staleness-reads), configure your app to use [`AS OF SYSTEM TIME`](https://docs.cockroachlabs.com/docs/stable/as-of-system-time) with the [`with_min_timestamp()` or `with_max_staleness()` function](https://docs.cockroachlabs.com/docs/stable/functions-and-operators#date-and-time-functions) whenever reading from the table. Note that only single-row point reads in single-statement (implicit) transactions are supported.

  ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
  SELECT code FROM postal_codes AS OF SYSTEM TIME with_max_staleness('10s') where id = 5;
  ```

## Characteristics

### Latency

#### Reads

Reads retrieve historical data from the closest replica and, therefore, never leave the region. This makes read latency very low but slightly stale.

For example, in the following diagram:

1. The read request in `us-central` reaches the regional load balancer.
2. The load balancer routes the request to a gateway node.
3. The gateway node routes the request to the closest replica for the table. In this case, the replica is *not* the leaseholder.
4. The replica retrieves the results as of your preferred staleness interval in the past and returns to the gateway node.
5. The gateway node returns the results to the client.

![Follower reads topology reads](https://mintcdn.com/cockroachlabs/iiBVxthghoAOO3UM/images/common/topology-patterns/topology_follower_reads_reads.png?fit=max\&auto=format\&n=iiBVxthghoAOO3UM\&q=85\&s=6589bb12122d56506eb981f2beb321df)

#### Writes

The replicas for the table are spread across all 3 regions, so writes involve multiple network hops across regions to achieve consensus. This increases write latency significantly.

For example, in the following animation:

1. The write request in `us-central` reaches the regional load balancer.
2. The load balancer routes the request to a gateway node.
3. The gateway node routes the request to the leaseholder replica for the table in `us-east`.
4. Once the leaseholder has appended the write to its Raft log, it notifies its follower replicas.
5. As soon as one follower has appended the write to its Raft log (and thus a majority of replicas agree based on identical Raft logs), it notifies the leaseholder and the write is committed on the agreeing replicas.
6. The leaseholder then returns acknowledgement of the commit to the gateway node.
7. The gateway node returns the acknowledgement to the client.

![Follower reads topology writes](https://mintcdn.com/cockroachlabs/iiBVxthghoAOO3UM/images/common/topology-patterns/topology_follower_reads_writes.gif?s=eff1e08de84c1419e996268c780bacc5)

### Resiliency

Because this pattern balances the replicas for the table across regions, one entire region can fail without interrupting access to the table:

![Follower reads topology region failure](https://mintcdn.com/cockroachlabs/iiBVxthghoAOO3UM/images/common/topology-patterns/topology_follower_reads_resiliency.png?fit=max\&auto=format\&n=iiBVxthghoAOO3UM\&q=85\&s=4a06918931322d43f48fa29bea6e352d)

## See also

* [Multi-Region Capabilities Overview](https://docs.cockroachlabs.com/docs/stable/multiregion-overview.md)
* [How to Choose a Multi-Region Configuration](https://docs.cockroachlabs.com/docs/stable/choosing-a-multi-region-configuration.md)
* [When to Use `ZONE` vs. `REGION` Survival Goals](https://docs.cockroachlabs.com/docs/stable/multiregion-survival-goals#when-to-use-zone-vs-region-survival-goals)
* [When to Use `REGIONAL` vs. `GLOBAL` Tables](https://docs.cockroachlabs.com/docs/stable/table-localities#when-to-use-regional-vs-global-tables)
* [Secondary regions](https://docs.cockroachlabs.com/docs/stable/multiregion-overview.md#secondary-regions)
* [`ALTER DATABASE ... SET SECONDARY REGION`](https://docs.cockroachlabs.com/docs/stable/alter-database#set-secondary-region)
* [`ALTER DATABASE ... DROP SECONDARY REGION`](https://docs.cockroachlabs.com/docs/stable/alter-database#drop-secondary-region)
* [Topology Patterns Overview](https://docs.cockroachlabs.com/docs/stable/topology-patterns.md)
  * Single-region patterns
    * [Basic Production](https://docs.cockroachlabs.com/docs/stable/topology-basic-production.md)
  * Multi-region patterns
    * [`REGIONAL` Tables](https://docs.cockroachlabs.com/docs/stable/regional-tables)
    * [`GLOBAL` Tables](https://docs.cockroachlabs.com/docs/stable/global-tables)
    * [Follower Reads](https://docs.cockroachlabs.com/docs/stable/topology-follower-reads.md)
    * [Follow-the-Workload](https://docs.cockroachlabs.com/docs/stable/topology-follow-the-workload.md)