A migration involves transferring data from a pre-existing source database onto a target CockroachDB cluster. Migrating data is a complex, multi-step process, and a data migration can take many different forms depending on your specific business and technical constraints.
Cockroach Labs provides a toolkit to aid in migrations.
This page provides an overview of the following:
Migration sequence
A migration to CockroachDB generally follows a variant of this sequence:
- Assess and discover: Inventory the source database, flag unsupported features, make a migration plan.
- Prepare the environment: Configure networking, users and permissions, bucket locations, and replication settings.
- Convert the source schema: Generate CockroachDB-compatible . Apply the converted schema to the target database. Drop constraints and indexes to facilitate data load.
- Load data into CockroachDB: Bulk load the source data into the CockroachDB cluster.
- Finalize target schema: Recreate indexes or constraints on CockroachDB that you previously dropped to facilitate data load.
- Replicate ongoing changes (optional): Keep CockroachDB in sync with the source. This may be necessary for migrations that minimize downtime.
- Stop application traffic: Limit user read/write traffic to the source database. This begins application downtime.
- Verify data consistency: Confirm that the CockroachDB data is consistent with the source.
- Enable failback (optional): Replicate data from the target back to the source, enabling a reversion to the source database in the event of migration failure.
- Cut over application traffic: Resume normal application use, with the CockroachDB cluster as the target database. This ends application downtime.
This sequence can vary depending on the needs on how your organization considers the migration variables. The common migration approaches describe some standard use cases, but even these may need to be modified to suit the needs of your migration.
The following diagram shows how the MOLT (Migrate Off Legacy Technology) toolkit is used at various stages of the migration sequence.
is a set of tools for schema conversion, data load, replication, and validation. Migrations with MOLT are resilient, restartable, and scalable to large data sets.
MOLT Fetch, Replicator, and Verify are CLI-based to maximize control, automation, and visibility during the data load and replication stages.
| Tool | Usage | Tested and supported sources | Release status |
|---|
| Schema Conversion Tool | Schema conversion | PostgreSQL, MySQL, Oracle, SQL Server | GA (Cloud only) |
| Fetch | Initial data load | PostgreSQL 11-16, MySQL 5.7-8.4, Oracle Database 19c (Enterprise Edition) and 21c (Express Edition) | GA |
| Replicator | Continuous replication | PostgreSQL 11-16, MySQL 5.7-8.4, Oracle Database 19c+, CockroachDB | GA |
| Verify | Schema and data validation | PostgreSQL 12-16, MySQL 5.7-8.4, Oracle Database 19c (Enterprise Edition) and 21c (Express Edition), CockroachDB | |
For CockroachDB-to-CockroachDB migrations, contact your account team for guidance.
- Identifies .
- Rewrites unsupported .
- Applies CockroachDB .
Fetch
performs the initial data load to CockroachDB. It supports:
- Multiple migration flows via
IMPORT INTO or COPY FROM.
- Data movement via .
- from multiple source tables and shards.
- .
- After exporting data with
IMPORT INTO, safe to retry failed or interrupted tasks from specific checkpoints.
Replicator
provides capabilities for minimal-downtime migrations. It supports:
- Continuous replication from source databases to CockroachDB.
- for balancing throughput and transactional guarantees.
- Failback replication from CockroachDB back to source databases.
- for high-throughput workloads.
- for defining data transformations.
Verify
checks for data and schema discrepancies between the source database and CockroachDB. It performs the following verifications:
- Table structure.
- Column definition.
- Row-level data.
Migration variables
You must decide how you want your migration to handle each of the following variables. These decisions will depend on your specific business and technical considerations. The MOLT toolkit supports any set of decisions made for the supported source databases.
| Variable | Description |
|---|
| Do you want to migrate all of your data at once, or do you want to split your data up into phases and migrate one phase at a time? |
| After the initial data load (or after the initial load of each phase), do you want to stream further changes to that data from the source to the target? |
| If there are discrepancies between the source and target schema, how will you define those data transformations, and when will those transformations occur? |
| How and when will you verify that the data in CockroachDB matches the source database? |
| What approach will you use to roll back the migration if issues arise during or after cutover? |
Common migration approaches
MOLT supports various migration flows using for data loading and for ongoing replication. These common approaches are variants of the general migration sequence.
| Migration approach | Description | Best for |
|---|
| Perform a one-time bulk load of source data into CockroachDB. | Simple migrations with planned downtime. |
| Divide your data into separate phases and bulk load each phase. | Larger migrations with planned downtime per phase. |
| Perform an initial data load, then replicate ongoing changes continuously. | Minimal-downtime migrations. |
| Divide your data into separate phases. For each phase, perform an initial data load, then replicate ongoing changes continuously. Enable failback replication. | Risk-averse migrations with minimal downtime per phase. |
See also
