> ## Documentation Index
> Fetch the complete documentation index at: https://cockroachlabs.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Phased Bulk Load Migration from PostgreSQL

export const InternalLink = ({version, path = "", children, ...props}) => {
  let detectedVersion = version || "stable";
  if (typeof window !== 'undefined' && !version) {
    const match = window.location.pathname.match(/\/docs\/([^/]+)/);
    if (match) {
      detectedVersion = match[1];
    }
  }
  const normalizedPath = path.startsWith("/") ? path.slice(1) : path;
  return <a href={`/docs/${detectedVersion}/${normalizedPath}`} {...props}>
      {children}
    </a>;
};

A <InternalLink version="molt" path="migration-approach-phased-bulk-load">*Phased Bulk Load Migration*</InternalLink> involves <InternalLink version="molt" path="migration-overview">migrating data to CockroachDB</InternalLink> in several phases. Data can be sliced per tenant, per service, per region, or per table to suit the needs of the migration. In this approach, you stop application traffic to the source database *only* for the tables in a particular slice of data. You then migrate that phase of data to the target cluster using <InternalLink version="molt" path="molt-fetch">MOLT Fetch</InternalLink> during a **downtime window**. Application traffic is then cut over to those target tables after schema finalization and data verification. This process is repeated for each phase of data.

* Data is migrated to the target <InternalLink version="molt" path="migration-considerations-granularity">in phases</InternalLink>.

* This approach does not utilize <InternalLink version="molt" path="migration-considerations-replication">continuous replication</InternalLink>.

* <InternalLink version="molt" path="migration-considerations-rollback">Rollback</InternalLink> is manual. If you wish to roll back before the target has received any writes that are not present on the source database, nothing needs to be done. If you wish to roll back after the target has received writes that are not present on the source database, you must manually replicate these new rows on the source.

This approach is comparable to the <InternalLink version="molt" path="migration-approach-classic-bulk-load">Classic Bulk Load Migration</InternalLink>, but dividing the data into multiple phases allows each downtime window to be shorter, and it allows each phase of the migration to be less complex. Depending on how you divide the data, it also may allow your downtime windows to affect only a subset of users. For example, dividing the data per region could mean that, when migrating the data from Region A, application usage in Region B may remain unaffected. This approach may increase overall migration complexity: its duration is longer, you will need to do the work of partitioning the data, and you will have a longer period when you run both the source and the target database concurrently.

This approach is best for databases that are too large to migrate all at once, internal tools, dev/staging environments, and production environments that can handle business disruption. It can only be performed if your system can handle downtime for each migration phase, and if your source database can easily be divided into the phases you need.

This page describes an example scenario. While the commands provided can be copy-and-pasted, they may need to be altered or reconsidered to suit the needs of your specific environment.

<img src="https://mintcdn.com/cockroachlabs/mEy4bzaxzRbvNkMw/images/molt/molt_phased_bulk_load_flow.svg?fit=max&auto=format&n=mEy4bzaxzRbvNkMw&q=85&s=1f3c5b889b36fcc6a704b290f652c308" alt="Phased Bulk Load Migration flow" style={{ maxWidth: "100%", display: "block", marginLeft: "auto", marginRight: "auto" }} width="456" height="326" data-path="images/molt/molt_phased_bulk_load_flow.svg" />

## Example scenario

You have a moderately-sized (500GB) database that provides the data store for a web application. You want to migrate the entirety of this database to a new CockroachDB cluster. You will divide this migration into four geographic regions (A, B, C, and D). You schedule a maintenance window for each region over four subsequent evenings, and you announce them to your users (per region) several weeks in advance.

The application runs on a Kubernetes cluster with an NGINX Ingress Controller.

**Estimated system downtime:** 4 hours per region.

## Before the migration

* Install the <InternalLink version="molt" path="molt-fetch-installation#installation">MOLT (Migrate Off Legacy Technology)</InternalLink> tools.
* Review the <InternalLink version="molt" path="molt-fetch-best-practices">MOLT Fetch</InternalLink> documentation.
* <InternalLink version="molt" path="migration-strategy#develop-a-migration-plan">Develop a migration plan</InternalLink> and <InternalLink version="molt" path="migration-strategy#prepare-for-migration">prepare for the migration</InternalLink>.
* **Recommended:** Perform a dry run of this full set of instructions in a development environment that closely resembles your production environment. This can help you get a realistic sense of the time and complexity it requires.
* Announce the maintenance window to your users on a per-region basis.
* Understand the prerequisites and limitations of the MOLT tools:

### Limitations

#### MOLT Fetch limitations

* Only tables with <InternalLink version="stable" path="primary-key">primary key</InternalLink> types of <InternalLink version="stable" path="int">`INT`</InternalLink>, <InternalLink version="stable" path="float">`FLOAT`</InternalLink>, or <InternalLink version="stable" path="uuid">`UUID`</InternalLink> can be sharded with <InternalLink version="molt" path="molt-fetch-best-practices#configure-the-source-database-and-connection">`--export-concurrency`</InternalLink>.

* `GEOMETRY` and `GEOGRAPHY` types are not supported.

* `OID LOB` types in PostgreSQL are not supported, although similar types like `BYTEA` are supported.

#### MOLT Verify limitations

* MOLT Verify compares 20,000 rows at a time by default, and row values can change between batches, potentially resulting in temporary inconsistencies in data. To configure the row batch size, use the `--row_batch_size` <InternalLink version="molt" path="molt-verify#flags">flag</InternalLink>.
* MOLT Verify checks for collation mismatches on <InternalLink version="stable" path="primary-key">primary key</InternalLink> columns. This may cause validation to fail when a <InternalLink version="stable" path="string">`STRING`</InternalLink> is used as a primary key and the source and target databases are using different <InternalLink version="stable" path="collate">collations</InternalLink>.
* MOLT Verify might give an error in case of schema changes on either the source or target database.
* <InternalLink version="stable" path="spatial-data-overview#spatial-objects">Geospatial types</InternalLink> cannot yet be compared.

## Step 1: Prepare the source database

In this step, you will:

* [Create a dedicated migration user on your source database](#create-migration-user-on-source-database).

### Create migration user on source database

Create a dedicated migration user (for example, `MIGRATION_USER`) on the source database. This user is responsible for reading data from source tables during the migration. You will pass this username in the [source connection string](#source-connection-string).

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
CREATE USER migration_user WITH PASSWORD 'password';
```

Grant the user privileges to connect, view schema objects, and select the tables you migrate.

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
GRANT CONNECT ON DATABASE migration_db TO migration_user;
GRANT USAGE ON SCHEMA migration_schema TO migration_user;
GRANT SELECT ON ALL TABLES IN SCHEMA migration_schema TO migration_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA migration_schema GRANT SELECT ON TABLES TO migration_user;
```

## Step 2: Prepare the target database

In this step, you will:

* [Provision and run a new CockroachDB cluster](#provision-a-cockroachdb-cluster).
* [Define the tables on the target cluster](#define-the-target-tables) to match those on the source.
* [Create a SQL user on the target cluster](#create-the-sql-user) with the necessary write permissions.

### Provision a CockroachDB cluster

Use one of the following options to create and run a new CockroachDB cluster. This is your migration **target**.

#### Option 1: Create a secure cluster locally

If you have the CockroachDB binary installed locally, you can manually deploy a multi-node, self-hosted CockroachDB cluster on your local machine.

Learn how to <InternalLink version="stable" path="secure-a-cluster">deploy a CockroachDB cluster locally</InternalLink>.

#### Option 2: Create a CockroachDB Self-Hosted cluster on AWS

You can manually deploy a multi-node, self-hosted CockroachDB cluster on Amazon's AWS EC2 platform, using AWS's managed load-balancing service to distribute client traffic.

Learn how to <InternalLink version="stable" path="deploy-cockroachdb-on-aws">deploy a CockroachDB cluster on AWS</InternalLink>.

#### Option 3: Create a CockroachDB Cloud cluster

CockroachDB Cloud is a fully-managed service run by Cockroach Labs, which simplifies the deployment and management of CockroachDB.

[Sign up for a CockroachDB Cloud account](https://cockroachlabs.cloud) and <InternalLink version="cockroachcloud" path="create-your-cluster">create a cluster</InternalLink> using <InternalLink version="cockroachcloud" path="free-trial">trial credits</InternalLink>.

### Define the target tables

Convert the source table definitions into CockroachDB-compatible equivalents. CockroachDB supports the PostgreSQL wire protocol and is largely <InternalLink version="stable" path="postgresql-compatibility#features-that-differ-from-postgresql">compatible with PostgreSQL syntax</InternalLink>.

* The source and target table definitions must **match**. Review <InternalLink version="molt" path="molt-fetch#type-mapping">Type mapping</InternalLink> to understand which source types can be mapped to CockroachDB types.

  For example, a PostgreSQL source table defined as `CREATE TABLE migration_schema.tbl (pk INT PRIMARY KEY);` must have a corresponding schema and table in CockroachDB:

  ```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
  CREATE SCHEMA migration_schema;
  CREATE TABLE migration_schema.tbl (pk INT PRIMARY KEY);
  ```

  * MOLT Fetch can automatically define matching CockroachDB tables using the <InternalLink version="molt" path="molt-fetch#handle-target-tables">`drop-on-target-and-recreate`</InternalLink> option.

  * If you define the target tables manually, review how MOLT Fetch handles <InternalLink version="molt" path="molt-fetch#mismatch-handling">type mismatches</InternalLink>. You can use the [MOLT Schema Conversion Tool](#schema-conversion-tool) to create matching table definitions.

* Every table **must** have an explicit primary key.

  <Danger>
    Avoid using sequential keys. To learn more about the performance issues that can result from their use, refer to the <InternalLink version="stable" path="sql-faqs#how-do-i-generate-unique-slowly-increasing-sequential-numbers-in-cockroachdb">guidance on indexing with sequential keys</InternalLink>. If a sequential key is necessary in your CockroachDB table, you must create it manually, after using <InternalLink version="molt" path="molt-fetch">MOLT Fetch</InternalLink> to load and replicate the data.
  </Danger>

* Review <InternalLink version="molt" path="molt-fetch">Transformations</InternalLink> to understand how computed columns and partitioned tables can be mapped to the target, and how target tables can be renamed.

* By default on CockroachDB, `INT` is an alias for `INT8`, which creates 64-bit signed integers. PostgreSQL and MySQL default to 32-bit integers. Depending on your source database or application requirements, you may need to change the integer size to `4`. For more information, refer to <InternalLink version="stable" path="int#considerations-for-64-bit-signed-integers">Considerations for 64-bit signed integers</InternalLink>.

#### Schema Conversion Tool

The <InternalLink version="cockroachcloud" path="migrations-page">MOLT Schema Conversion Tool</InternalLink> (SCT) converts source table definitions to CockroachDB-compatible syntax. It requires a free <InternalLink version="cockroachcloud" path="create-an-account">CockroachDB Cloud account</InternalLink>.

1. Upload a source `.sql` file to convert the syntax and identify <InternalLink version="molt" path="migration-strategy#unimplemented-features-and-syntax-incompatibilities">unimplemented features and syntax incompatibilities</InternalLink> in the table definitions.

2. Import the converted table definitions to a CockroachDB cluster:
   * When migrating to CockroachDB Cloud, the Schema Conversion Tool automatically <InternalLink version="cockroachcloud" path="migrations-page#migrate-the-schema">applies the converted table definitions to a new Cloud database</InternalLink>.
   * When migrating to a self-hosted CockroachDB cluster, <InternalLink version="cockroachcloud" path="migrations-page#export-the-schema">export a converted DDL file</InternalLink> and pipe the <InternalLink version="stable" path="sql-statements#data-definition-statements">data definition language (DDL)</InternalLink> directly into <InternalLink version="stable" path="cockroach-sql">`cockroach sql`</InternalLink>.

#### Drop constraints and indexes

To optimize data load performance, drop all non-`PRIMARY KEY` <InternalLink version="stable" path="alter-table#drop-constraint">constraints</InternalLink> and <InternalLink version="stable" path="drop-index">indexes</InternalLink> on the target CockroachDB database before migrating:

* <InternalLink version="stable" path="foreign-key">`FOREIGN KEY`</InternalLink>
* <InternalLink version="stable" path="unique">`UNIQUE`</InternalLink>
* <InternalLink version="stable" path="schema-design-indexes">Secondary indexes</InternalLink>
* <InternalLink version="stable" path="check">`CHECK`</InternalLink>
* <InternalLink version="stable" path="default-value">`DEFAULT`</InternalLink>
* <InternalLink version="stable" path="not-null">`NOT NULL`</InternalLink> (you do not need to drop this constraint when using `drop-on-target-and-recreate` for [table handling](#table-handling-mode))

<Danger>
  Do **not** drop <InternalLink version="stable" path="primary-key">`PRIMARY KEY`</InternalLink> constraints.
</Danger>

You can [recreate the constraints and indexes after loading the data](#add-constraints-and-indexes).

### Create the SQL user

Create a SQL user in the CockroachDB cluster that has the necessary privileges.

To create a user `crdb_user` in the default database (you will pass this username in the [target connection string](#target-connection-string)):

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
CREATE USER crdb_user WITH PASSWORD 'password';
```

Grant database-level privileges for schema creation within the target database:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
GRANT ALL ON DATABASE defaultdb TO crdb_user;
```

Grant permission to modify cluster settings:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
GRANT SYSTEM MODIFYCLUSTERSETTING TO crdb_user;
```

Grant usage on schemas being migrated:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
GRANT USAGE ON SCHEMA migration_schema TO crdb_user;
GRANT USAGE ON SCHEMA public TO crdb_user;
```

Grant user privileges to create tables in the `migration_schema` schema and internal MOLT tables like `_molt_fetch_exceptions` in the `public` CockroachDB schema:

<Note>
  Ensure that you are connected to the target database.
</Note>

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
GRANT CREATE ON SCHEMA migration_schema TO crdb_user;
GRANT CREATE ON SCHEMA public TO crdb_user;
```

If you manually defined the target tables (which means that [`drop-on-target-and-recreate`](#table-handling-mode) will not be used), grant the following privileges on the schema:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA migration_schema TO crdb_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA migration_schema
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO crdb_user;
```

Grant the same privileges for internal MOLT tables in the `public` CockroachDB schema:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO crdb_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO crdb_user;
```

If you will be running Fetch with [`drop-on-target-and-recreate`](#table-handling-mode), and the target schema has pre-existing tables that were created by a different user, you may need to change table ownership to allow the migration user to drop those tables. Make the following alteration for each table:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ALTER TABLE migration_schema.table1 OWNER TO crdb_user;
```

Depending on the MOLT Fetch <InternalLink version="molt" path="molt-fetch">data load mode</InternalLink> you will use, grant the necessary privileges to run either [`IMPORT INTO`](#import-into-privileges) or [`COPY FROM`](#copy-from-privileges) on the target tables:

#### `IMPORT INTO` privileges

Grant `SELECT`, `INSERT`, and `DROP` (required because the table is taken offline during the `IMPORT INTO`) privileges on all tables being migrated:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
GRANT SELECT, INSERT, DROP ON ALL TABLES IN SCHEMA migration_schema TO crdb_user;
```

If you plan to use [cloud storage with implicit authentication](#cloud-storage-authentication) for data load, grant the `EXTERNALIOIMPLICITACCESS` <InternalLink version="stable" path="security-reference/authorization#supported-privileges">system-level privilege</InternalLink>:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
GRANT SYSTEM EXTERNALIOIMPLICITACCESS TO crdb_user;
```

#### `COPY FROM` privileges

Grant <InternalLink version="stable" path="security-reference/authorization#admin-role">`admin`</InternalLink> privileges to the user:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
GRANT admin TO crdb_user;
```

#### Set session variable

Ensure that the `statement_timeout` <InternalLink path="set-vars">session variable</InternalLink> is set to `0s` for the user:

```sql theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
ALTER ROLE crdb_user SET statement_timeout = '0s';
```

## Migrate each phase

Steps 3-7 are run for each phase of the data migration. Within the first migration downtime window, you will run through these steps for Region A. You will repeat these steps for the other regions during each subsequent downtime window.

## Step 3: Stop application traffic

With both the source and target databases prepared for the data load, it's time to stop application traffic to the source for a particular region.

If the Kubernetes cluster that deploys the application has pre-region deployments (for example, `app-us`, `app-eu`, `app-apac`), you can scale down only the deployment for that region.

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl scale deploy/app-eu --replicas=0
```

Or this can be handled by the NGINX Ingress Controller, by including the following to your NGINX configuration, ensuring that the conditional statement is suitable for your deployment:

```yml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      if ($http_x_region = "eu") {
        return 503;
      }
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app
            port:
              number: 80
```

<Danger>
  Application downtime begins now, for users in the given region.

  It is strongly recommended that you perform a dry run of this migration in a test environment. This will allow you to practice using the MOLT tools in real time, and it will give you an accurate sense of how long application downtime might last.
</Danger>

## Step 4: Load data into CockroachDB

In this step, you will:

* [Configure MOLT Fetch with the flags needed for your migration](#configure-molt-fetch).
* [Run MOLT Fetch](#run-molt-fetch).
* [Understand how to continue a load after an interruption](#continue-molt-fetch-after-an-interruption).

### Configure MOLT Fetch

The <InternalLink version="molt" path="molt-fetch">MOLT Fetch documentation</InternalLink> includes detailed information about how to <InternalLink version="molt" path="molt-fetch">configure MOLT Fetch</InternalLink>, and how to <InternalLink version="molt" path="molt-fetch-monitoring">monitor MOLT Fetch metrics</InternalLink>.

When you run `molt fetch`, you can configure the following options for data load:

<a id="schema-and-table-filtering" />

<a id="source-connection-string" />

<a id="table-handling-mode" />

<a id="target-connection-string" />

<a id="cloud-storage-authentication" />

<a id="secure-connections" />

<a id="intermediate-file-storage" />

<a id="data-load-mode" />

<a id="connection-strings" />

* <InternalLink version="molt" path="molt-fetch#specify-source-and-target-databases">Specify source and target databases</InternalLink>: Specify URL‑encoded source and target connections.
* <InternalLink version="molt" path="molt-fetch#select-data-to-migrate">Select data to migrate</InternalLink>: Specify schema and table names to migrate. **Important for a phased migration.**
* <InternalLink version="molt" path="molt-fetch#define-intermediate-storage">Define intermediate file storage</InternalLink>: Export data to cloud storage or a local file server.
* <InternalLink version="molt" path="molt-fetch#define-fetch-mode">Define fetch mode</InternalLink>: Specifies whether data will only be loaded into/from intermediate storage.
* <InternalLink version="molt" path="molt-fetch#shard-tables-for-concurrent-export">Shard tables</InternalLink>: Divide larger tables into multiple shards during data export.
* <InternalLink version="molt" path="molt-fetch#import-into-vs-copy-from">Data load mode</InternalLink>: Choose between `IMPORT INTO` and `COPY FROM`.
* <InternalLink version="molt" path="molt-fetch#handle-target-tables">Table handling mode</InternalLink>: Determine how existing target tables are initialized before load.
* <InternalLink version="molt" path="molt-fetch#define-transformations">Define data transformations</InternalLink>: Define any row-level transformations to apply to the data before it reaches the target.
* <InternalLink version="molt" path="molt-fetch-monitoring">Monitor fetch metrics</InternalLink>: Configure metrics collection during initial data load.

Read through the documentation to understand how to configure your `molt fetch` command and its flags. Follow <InternalLink version="molt" path="molt-fetch-best-practices">best practices</InternalLink>, especially those related to security.

At minimum, the `molt fetch` command should include the source, target, data path, and <InternalLink version="molt" path="molt-fetch-commands-and-flags">`--ignore-replication-check`</InternalLink> flags. For a phased migration, you may also choose to include <InternalLink version="molt" path="molt-fetch-commands-and-flags">`--schema-filter`</InternalLink> or <InternalLink version="molt" path="molt-fetch-commands-and-flags">`--table-filter`</InternalLink> flags:

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
molt fetch \
--source $SOURCE \
--target $TARGET \
--table-filter '.*user.*' \
--bucket-path 's3://bucket/path' \
--ignore-replication-check
```

However, depending on the needs of your migration, you may have many more flags set, and you may need to prepare some accompanying .json files.

### Run MOLT Fetch

Perform the bulk load of the source data.

1. Run the <InternalLink version="molt" path="molt-fetch">MOLT Fetch</InternalLink> command to move the source data into CockroachDB. This example command passes the source and target connection strings [as environment variables](#secure-connections), writes [intermediate files](#intermediate-file-storage) to S3 storage, and uses the `truncate-if-exists` [table handling mode](#table-handling-mode) to truncate the target tables before loading data. It limits the migration to a single schema and filters for three specific tables. The <InternalLink version="molt" path="molt-fetch">data load mode</InternalLink> defaults to `IMPORT INTO`. Include the `--ignore-replication-check` flag to skip replication checkpoint queries, which eliminates the need to configure the source database for logical replication.

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   molt fetch \
   --source $SOURCE \
   --target $TARGET \
   --schema-filter 'migration_schema' \
   --table-filter 'employees|payments|orders' \
   --bucket-path 's3://migration/data/cockroach' \
   --table-handling truncate-if-exists \
   --ignore-replication-check
   ```

2. Check the output to observe `fetch` progress.

   A `starting fetch` message indicates that the task has started:

   ```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   {"level":"info","type":"summary","num_tables":3,"cdc_cursor":"0/43A1960","time":"2025-02-10T14:28:11-05:00","message":"starting fetch"}
   ```

   `data extraction` messages are written for each table that is exported to the location in `--bucket-path`:

   ```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   {"level":"info","table":"migration_schema.employees","time":"2025-02-10T14:28:11-05:00","message":"data extraction phase starting"}
   ```

   ```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   {"level":"info","table":"migration_schema.employees","type":"summary","num_rows":200000,"export_duration_ms":1000,"export_duration":"000h 00m 01s","time":"2025-02-10T14:28:12-05:00","message":"data extraction from source complete"}
   ```

   `data import` messages are written for each table that is loaded into CockroachDB:

   ```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   {"level":"info","table":"migration_schema.employees","time":"2025-02-10T14:28:12-05:00","message":"starting data import on target"}
   ```

   ```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   {"level":"info","table":"migration_schema.employees","type":"summary","net_duration_ms":1899.748333,"net_duration":"000h 00m 01s","import_duration_ms":1160.523875,"import_duration":"000h 00m 01s","export_duration_ms":1000,"export_duration":"000h 00m 01s","num_rows":200000,"cdc_cursor":"0/43A1960","time":"2025-02-10T14:28:13-05:00","message":"data import on target for table complete"}
   ```

   A `fetch complete` message is written when the fetch task succeeds:

   ```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   {"level":"info","type":"summary","fetch_id":"f5cb422f-4bb4-4bbd-b2ae-08c4d00d1e7c","num_tables":3,"tables":["migration_schema.employees","migration_schema.payments","migration_schema.payments"],"skipped_unmigratable_tables":[],"cdc_cursor":"0/3F41E40","net_duration_ms":6752.847625,"net_duration":"000h 00m 06s","time":"2024-03-18T12:30:37-04:00","message":"fetch complete"}
   ```

### Continue MOLT Fetch after an interruption

If MOLT Fetch fails while loading data into CockroachDB from intermediate files, it exits with an error message, fetch ID, and *continuation token* for each table that failed to load on the target database.

```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
{"level":"info","table":"public.employees","file_name":"shard_01_part_00000001.csv.gz","message":"creating or updating token for duplicate key value violates unique constraint \"employees_pkey\"; Key (id)=(1) already exists."}
{"level":"info","table":"public.employees","continuation_token":"a1b2c3d4-e5f6-7890-abcd-ef1234567890","message":"created continuation token"}
{"level":"info","fetch_id":"f5cb422f-4bb4-4bbd-b2ae-08c4d00d1e7c","message":"continue from this fetch ID"}
{"level":"error","message":"Error: error from fetching table for public.employees: error importing data: duplicate key value violates unique
  constraint \"employees_pkey\" (SQLSTATE 23505)"}
```

You can use this information to <InternalLink version="molt" path="molt-fetch">continue the task from the *continuation point*</InternalLink> where it was interrupted.

Continuation is only possible under the following conditions:

* All data has been exported from the source database into intermediate files on <InternalLink version="molt" path="molt-fetch#bucket-path">cloud</InternalLink> or <InternalLink version="molt" path="molt-fetch#local-path">local storage</InternalLink>.
* The *initial load* of source data into the target CockroachDB database is incomplete.
* The load uses [`IMPORT INTO` rather than `COPY FROM`](#data-load-mode).

<Note>
  Only one fetch ID and set of continuation tokens, each token corresponding to a table, are active at any time. See <InternalLink version="molt" path="molt-fetch#list-active-continuation-tokens">List active continuation tokens</InternalLink>.
</Note>

The following command reattempts the data load starting from a specific continuation file, but you can also use individual continuation tokens to <InternalLink version="molt" path="molt-fetch">reattempt the data load for individual tables</InternalLink>.

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
molt fetch \
--source $SOURCE \
--target $TARGET \
--schema-filter 'migration_schema' \
--table-filter 'employees|payments|orders' \
--bucket-path 's3://migration/data/cockroach' \
--table-handling truncate-if-exists \
--ignore-replication-check \
--fetch-id f5cb422f-4bb4-4bbd-b2ae-08c4d00d1e7c \
--continuation-file-name shard_01_part_00000001.csv.gz
```

## Step 5: Verify the data

In this step, you will use <InternalLink version="molt" path="molt-verify">MOLT Verify</InternalLink> to confirm that the source and target data is consistent. This ensures that the data load was successful. Use MOLT Verify's <InternalLink version="molt" path="molt-verify#flags">`--schema-filter`</InternalLink> or <InternalLink version="molt" path="molt-verify#flags">`--table-filter`</InternalLink> to select only the tables that are relevant for the given phase.

### Run MOLT Verify

1. Run the <InternalLink version="molt" path="molt-verify">MOLT Verify</InternalLink> command, specifying the source and target [connection strings](#connection-strings) and the tables to validate.

   ```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   molt verify \
   --source $SOURCE \
   --target $TARGET \
   --table-filter 'employees|payments|orders'
   ```

2. Check the output to observe `verify` progress.

   A `verification in progress` indicates that the task has started:

   ```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   {"level":"info","time":"2025-02-10T15:35:04-05:00","message":"verification in progress"}
   ```

   `starting verify` messages are written for each specified table:

   ```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   {"level":"info","time":"2025-02-10T15:35:04-05:00","message":"starting verify on public.employees, shard 1/1"}
   ```

   A `finished row verification` message is written after each table is compared. If `num_success` equals `num_truth_rows` and the error counters (`num_missing`, `num_mismatch`, `num_extraneous`, and `num_column_mismatch`) are all `0`, the table verified successfully. Any non-zero values in the error counters indicate data discrepancies that need investigation. For details on each field, refer to the <InternalLink version="molt" path="molt-verify#usage">MOLT Verify</InternalLink> page.

   ```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   {"level":"info","type":"summary","table_schema":"public","table_name":"employees","num_truth_rows":200004,"num_success":200004,"num_conditional_success":0,"num_missing":0,"num_mismatch":0,"num_extraneous":0,"num_live_retry":0,"num_column_mismatch":0,"time":"2025-02-10T15:35:05-05:00","message":"finished row verification on public.employees (shard 1/1)"}
   ```

   A `verification complete` message is written when the verify task succeeds:

   ```json theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
   {"level":"info","net_duration_ms":699.804875,"net_duration":"000h 00m 00s","time":"2025-02-10T15:35:05-05:00","message":"verification complete"}
   ```

## Step 6: Finalize the target schema

### Add constraints and indexes

Add any constraints or indexes that you previously [removed from the CockroachDB schema](#drop-constraints-and-indexes) to facilitate data load.

<Note>
  If you used the `--table-handling drop-on-target-and-recreate` option for data load, only <InternalLink version="stable" path="primary-key">`PRIMARY KEY`</InternalLink> and <InternalLink version="stable" path="not-null">`NOT NULL`</InternalLink> constraints are preserved. You **must** manually recreate all other constraints and indexes.
</Note>

For the appropriate SQL syntax, refer to <InternalLink version="stable" path="alter-table#add-constraint">`ALTER TABLE ... ADD CONSTRAINT`</InternalLink> and <InternalLink version="stable" path="create-index">`CREATE INDEX`</InternalLink>. Review the <InternalLink version="stable" path="schema-design-indexes#best-practices">best practices for creating secondary indexes</InternalLink> on CockroachDB.

## Step 7: Cut over application traffic

With the target cluster verified and finalized, it's time to resume application traffic for the current migration phase.

### Modify application code

In the application back end, update the application to route traffic for this migration phase to the CockroachDB cluster. A simple example:

```yml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
env:
  - name: DATABASE_URL_US_EAST
    value: postgres://root@cockroachdb.us-east:26257/defaultdb?sslmode=verify-full
  - name: DATABASE_URL_US_WEST
    value: postgres://legacy-db.us-west:5432/defaultdb  # Still on source
```

In your application code, route database connections based on the user's region:

```python theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
def get_db_connection(user_region):
    if user_region == "us-east":
        return os.getenv("DATABASE_URL_US_EAST")  # CockroachDB
    else:
        return os.getenv("DATABASE_URL_US_WEST")  # Source database
```

### Resume application traffic

If you halted traffic by scaling down a regional Kubernetes deployment, scale it back up.

```shell theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
kubectl scale deploy/app-eu --replicas=3
```

Or if this was handled by the NGINX Controller, remove the 503 block that was written in step 3:

```yml theme={"theme":{"light":"catppuccin-mocha","dark":"catppuccin-mocha"}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app
#   annotations:
#     nginx.ingress.kubernetes.io/server-snippet: |
#       if ($http_x_region = "eu") {
#         return 503;
#       }
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app
            port:
              number: 80
```

This ends downtime for the current migration phase.

## Repeat for each phase

During the next scheduled, regional downtime window, [return to step 3](#step-3-stop-application-traffic) to migrate the next phase of data. Repeat steps 3-7 for each phase of data, until every region's data has been migrated and all application traffic has been cut over to the target.

## See also

* <InternalLink version="molt" path="migration-overview">Migration Overview</InternalLink>
* <InternalLink version="molt" path="migration-considerations">Migration Considerations</InternalLink>
* <InternalLink version="molt" path="migration-approach-classic-bulk-load">Classic Bulk Load Migration</InternalLink>
* <InternalLink version="molt" path="molt-fetch">MOLT Fetch</InternalLink>
* <InternalLink version="molt" path="molt-verify">MOLT Verify</InternalLink>
* <InternalLink version="molt" path="molt-fetch-troubleshooting">MOLT Fetch Troubleshooting for PostgreSQL</InternalLink>
* <InternalLink version="molt" path="molt-replicator-troubleshooting">MOLT Replicator Troubleshooting for PostgreSQL</InternalLink>
