MOLT Replicator

On this page

MOLT Replicator continuously replicates changes from source databases to CockroachDB as part of a database migration. It supports live ongoing migrations to CockroachDB from a source database, and enables backfill from CockroachDB to your source database for failback scenarios to preserve a rollback option during a migration window.

MOLT Replicator consumes change data from CockroachDB changefeeds, PostgreSQL logical replication streams, MySQL GTID-based replication, and Oracle LogMiner. It applies changes to target databases while maintaining configurable consistency , and features an embedded TypeScript/JavaScript environment for configuration and live data transforms.

Supported databases

The following source databases are supported:

• CockroachDB
• PostgreSQL 11-16
• MySQL 5.7, 8.0 and later
• Oracle Database 19c (Enterprise Edition) and 21c (Express Edition)

The following target databases are supported:

• CockroachDB
• PostgreSQL
• MySQL
• Oracle

Installation

To install MOLT, download the binary that matches your architecture and source database:

The following binaries are included:

• molt
• replicator

Both molt and replicator must be in your current working directory.

To display the current version of each binary, run molt --version and replicator --version.

For previous binaries, refer to the MOLT version manifest. For release details, refer to the MOLT changelog.

Docker images

MOLT Replicator

Docker images for MOLT Replicator are also available as a standalone binary:

docker pull cockroachdb/replicator

To pull a specific version (for example, v1.1.1):

docker pull cockroachdb/replicator:v1.1.1

For details on running in Docker, refer to Docker usage.

Setup

Complete the following items before using MOLT Replicator:

• Follow the recommendations in Best practices.

• Configure the source database for replication based on your source type. Refer to the migration workflow tutorials for PostgreSQL, MySQL, Oracle, and CockroachDB sources.

• Ensure that the SQL user running MOLT Replicator has appropriate privileges on the source and target databases, as described in the migration workflow tutorials.

Docker usage

For details on pulling Docker images, refer to Docker images.

Performance

MOLT Fetch, Verify, and Replicator are likely to run more slowly in a Docker container than on a local machine. To improve performance, increase the memory or compute resources, or both, on your Docker container.

Local connection strings

When testing locally, specify the host as follows:

• For macOS, use host.docker.internal. For example:

  --source 'postgres://postgres:postgres@host.docker.internal:5432/molt?sslmode=disable'
  --target "postgres://root@host.docker.internal:26257/molt?sslmode=disable"
  

• For Linux and Windows, use 172.17.0.1. For example:

  --source 'postgres://postgres:postgres@172.17.0.1:5432/molt?sslmode=disable'
  --target "postgres://root@172.17.0.1:26257/molt?sslmode=disable"
  

How it works

Failback from CockroachDB (start): MOLT Replicator acts as an HTTP webhook sink for a CockroachDB changefeed. To avoid mixing streams of incoming data, only one changefeed should point to Replicator at a time. Replicator receives mutations from source cluster nodes, can optionally buffer them in a CockroachDB staging cluster, and then applies time-ordered transactional batches to the target database. Mutations are applied as UPSERT or DELETE statements while respecting foreign-key and table dependencies.

PostgreSQL source (pglogical): MOLT Replicator uses PostgreSQL logical replication, which is based on publications and replication slots. You create a publication for the target tables, and a slot marks consistent replication points. MOLT Replicator consumes this logical feed directly and applies the data in sorted batches to the target.

MySQL source (mylogical): MOLT Replicator relies on MySQL GTID-based replication to read change data from MySQL binlogs. It works with MySQL versions that support GTID-based replication and applies transactionally consistent feeds to the target. Binlog features that do not use GTIDs are not supported.

Oracle source (oraclelogminer): MOLT Replicator uses Oracle LogMiner to capture change data from Oracle redo logs, supporting replication for a single database user/schema. Both multitenant (CDB) and non-CDB Oracle architectures are supported. Replicator periodically queries LogMiner-populated views and processes transactional data in ascending SCN windows for reliable throughput while maintaining consistency.

Consistency modes

MOLT Replicator offers three consistency modes for balancing throughput and transactional guarantees:

1. Consistent (default for CockroachDB sources): Preserves per-row order and source transaction atomicity. Concurrent transactions are controlled by --parallelism.

2. BestEffort: Relaxes atomicity across tables that do not have foreign key constraints between them (maintains coherence within FK-connected groups). Enable with --bestEffortOnly or allow auto-entry via --bestEffortWindow set to a positive duration (e.g., 1s).

3. Immediate (default for PostgreSQL, MySQL, and Oracle sources): Applies updates as they arrive to Replicator with no buffering or waiting for resolved timestamps. Provides highest throughput but requires no foreign keys on the target schema.

Commands

Flags

Replication flags

PostgreSQL replication flags

The following flags are used when replicating from a PostgreSQL source database.

MySQL replication flags

The following flags are used when replicating from a MySQL source database.

Oracle replication flags

The following flags are used when replicating from an Oracle source database.

Failback replication flags

The following flags are used for failback from CockroachDB.

Usage

The following sections describe how to use MOLT Replicator with different source database types.

Source and target databases

--sourceConn

--sourceConn specifies the connection string of the source database.

CockroachDB:

--sourceConn 'postgresql://{username}:{password}@{host}:{port}/{database}'

PostgreSQL:

--sourceConn 'postgresql://{username}:{password}@{host}:{port}/{database}'

MySQL:

--sourceConn 'mysql://{username}:{password}@{protocol}({host}:{port})/{database}'

Oracle:

--sourceConn 'oracle://{username}:{password}@{host}:{port}/{service_name}'

In Oracle migrations, the --sourceConn connection string specifies a CDB and the --sourcePDBConn connection string specifies a PDB (in Oracle Multitenant databases). The {username} corresponds to the owner of the tables you will migrate.

--targetConn

--targetConn specifies the target database connection string:

CockroachDB and PostgreSQL:

--targetConn 'postgresql://{username}:{password}@{host}:{port}/{database}'

MySQL:

--targetConn 'mysql://{username}:{password}@{protocol}({host}:{port})/{database}'

Oracle:

--targetConn 'oracle://{username}:{password}@{host}:{port}/{service_name}'

Data transformations

Use --userscript to specify the path to a TypeScript/JavaScript file for data transformations and filtering. The userscript provides an embedded environment for configuration and live data transforms during replication.

Resume replication

MOLT Replicator can resume replication if it is interrupted. Specify the staging schema with the --stagingSchema flag. MOLT Replicator uses the schema as a replication marker for streaming changes.

--stagingSchema {schema_name}

You must include the --stagingSchema flag when resuming replication, as the schema provides checkpointing data that enables replication to resume from the correct point.

For detailed steps, refer to Resume replication.

Staging schema

The staging schema stores replication metadata, checkpoint information, and buffered mutations during replication. MOLT Replicator uses this schema to:

• Track replication progress: Store checkpoints that enable resuming from the correct point after interruptions
• Buffer mutations: Temporarily store data changes before applying them to the target in transaction order
• Maintain consistency: Ensure MVCC time-ordered transactional batches while respecting table dependencies
• Enable recovery: Provide restart capabilities with robust failure handling

For fresh replication runs, clean out the staging database memo table to avoid conflicts with previous checkpoints. Use --stagingCreateSchema to automatically create the staging schema if it doesn't exist.

Best practices

Test and validate

To verify that your connections and configuration work properly, run MOLT Replicator in a staging environment before replicating any data in production. Use a test or development environment that closely resembles production.

Connection strings

URL-encode the connection strings for the source and target databases. This ensures that the MOLT tools can parse special characters in your password.

To keep your database credentials out of shell history and logs, follow these best practices when specifying your source and target connection strings:

• Avoid plaintext connection strings.

• URL-encode connection strings for the source database and CockroachDB so special characters in passwords are handled correctly.

  • Given a password a$52&, pass it to the molt escape-password command with single quotes:

    molt escape-password --password 'a$52&'
    

    Use the encoded password in your --source connection string. For example:

    --source 'postgres://migration_user:a%2452%26@localhost:5432/replicationload'
    

• Provide your connection strings as environment variables. For example:

  export SOURCE="postgres://migration_user:a%2452%26@localhost:5432/molt?sslmode=verify-full"
  export TARGET="postgres://root@localhost:26257/molt?sslmode=verify-full"
  

  Afterward, reference the environment variables as follows:

  --source $SOURCE
  --target $TARGET
  

• If possible, use an external secrets manager to load the environment variables from stored secrets.

Optimize performance

Configure the following replicator flags to optimize replication throughput and resource usage. Test different combinations in a staging environment to find the optimal balance of stability and performance for your workload.

Monitoring and observability

• Monitor lag metrics (source_lag_seconds, target_lag_seconds) and apply/stage duration/error counters to track replication performance. Set up Prometheus and Grafana to poll the metrics and view them in a dashboard. Use the published Grafana dashboard to visualize these metrics.

• Pay close attention to warning and error level logging, as it indicates when Replicator is misbehaving. Enable trace logging with -vv for additional visibility when troubleshooting.

By default, MOLT Replicator exports Prometheus metrics at the address specified by --metricsAddr (default :30005) at the path /_/varz. For example: http://localhost:30005/_/varz.

Cockroach Labs recommends monitoring the following metrics during replication:

You can use the Replicator Grafana dashboard to visualize these metrics. For Oracle-specific metrics, import this Oracle Grafana dashboard.

To check MOLT Replicator health, run curl http://localhost:30005/_/healthz. This returns a status code of 200 if Replicator is running.

Logging

By default, MOLT Replicator writes two streams of logs: operational logs to stdout (including warning, info, trace, and some errors) and final errors to stderr.

Redirect both streams properly to ensure all logs are captured for troubleshooting:

# Merge both streams to console
./replicator ... 2>&1

# Redirect both streams to a file
./replicator ... > output.log 2>&1

# Merge streams to console while saving to file
./replicator > >(tee replicator.log) 2>&1

# Use logDestination flag to write all logs to a file
./replicator --logDestination replicator.log ...

Security

Cockroach Labs strongly recommends the following:

Connection security and credentials

• Use TLS-enabled connection strings for --sourceConn, --targetConn, and --stagingConn parameters. Remove sslmode=disable from production connection strings.
• For webhook endpoints (failback scenarios), configure server certificates using --tlsCertificate and --tlsPrivateKey to specify the certificate and private key file paths.
• Configure proper TLS certificates in CockroachDB changefeed webhook URLs instead of using insecure_tls_skip_verify=true.
• If relevant, enable JWT authentication for additional security.

CockroachDB changefeeds

For failback scenarios, generate TLS certificates using self-signed certificates, certificate authorities like Let's Encrypt, or your organization's certificate management system.

Encode certificates for changefeed webhook URLs:

• Webhook URLs: Use both URL encoding and base64 encoding: base64 -i ./client.crt | jq -R -r '@uri'
• Non-webhook contexts: Use base64 encoding only: base64 -w 0 ca.cert

Client certificates in changefeed webhook URLs must correspond to server certificates in MOLT Replicator configuration to ensure proper TLS handshake.

Production considerations

• Avoid --disableAuthentication and --tlsSelfSigned flags in production environments. These flags should only be used for testing or development purposes.

Examples

For detailed examples of using MOLT Replicator usage, refer to the migration workflow tutorials:

• Load and replicate: Load data with MOLT Fetch and set up ongoing replication with MOLT Replicator.
• Resume Replication: Resume replication to CockroachDB after an interruption.
• Migration failback: Replicate changes from CockroachDB back to the source database.

See also

• Migration Overview
• Migration Strategy
• MOLT Fetch
• Load and replicate
• Resume Replication
• Migration Failback