Skip to main content
This page provides quick setup guides for connecting changefeeds to sinks and for using sinkless changefeeds. For a summary of changefeed features, refer to the page. Changefeeds can emit messages to the following sinks: Refer to the page for more detail on forming sink URIs, available sink query parameters, and specifics on configuration. Cockroach Labs recommends monitoring your changefeeds to track and usage. Refer to the page for more information.
Before you run the examples, verify that you have the CHANGEFEED privilege in order to create and manage changefeed jobs. Refer to for more details.You can create an external connection to represent a changefeed sink URI. This allows you to specify the external connection’s name in statements rather than the provider-specific URI. For detail on using external connections, see the page.

​
Create a changefeed connected to Kafka

In this example, you’ll set up a changefeed for a single-node cluster that is connected to a Kafka sink. The changefeed will watch two tables.
  1. Use the command to start a single-node cluster:
  2. Download and extract the Confluent platform (which includes Kafka).
  3. In a new terminal window, go to the extracted confluent-<version directory and start Confluent:
    Only zookeeper and kafka are needed. To troubleshoot Confluent, see their docs and the Quick Start Guide.
  4. Create two Kafka topics:
You are expected to create any Kafka topics with the necessary number of replications and partitions. Topics can be created manually or Kafka brokers can be configured to automatically create topics with a default partition count and replication factor.
  1. As the root user, open the :
  2. Enable the kv.rangefeed.enabled :
If you are working on a CockroachDB Standard or Basic cluster, the kv.rangefeed.enabled cluster setting is enabled by default.
  1. Create a database called cdc_demo:
  2. Set the database as the default:
  3. Create a table and add data:
  4. Create another table and add data:
  5. Start the changefeed:
    This will start up the changefeed in the background and return the job_id. The changefeed writes to Kafka.
  6. In a new terminal, move into the extracted confluent-<version directory and start watching the Kafka topics:
    The initial scan displays the state of the tables as of when the changefeed started (therefore, the initial value of "Petee" is omitted).
This example only prints the value. To print both the key and value of each message in the changefeed (e.g., to observe what happens with DELETEs), use the --property print.key=true flag.
  1. Back in the SQL client, insert more data:
  2. Back in the terminal where you’re watching the Kafka topics, the following output has appeared:
  3. When you are done, exit the SQL shell (\q).
  4. To stop cockroach: Get the process ID of the node:
    Gracefully shut down the node, specifying its process ID:
  5. To stop Kafka, move into the extracted confluent-<version directory and stop Confluent:

​
Create a changefeed connected to Kafka using Avro

In this example, you’ll set up a changefeed for a single-node cluster that is connected to a Kafka sink and emits Avro records. The changefeed will watch two tables.
  1. Use the command to start a single-node cluster:
  2. Download and extract the Confluent platform (which includes Kafka).
  3. Move into the extracted confluent-<version directory and start Confluent:
    Only zookeeper, kafka, and schema-registry are needed. To troubleshoot Confluent, see their docs and the Quick Start Guide.
  4. Create two Kafka topics:
    You are expected to create any Kafka topics with the necessary number of replications and partitions. Topics can be created manually or Kafka brokers can be configured to automatically create topics with a default partition count and replication factor.
  5. As the root user, open the :
  6. Enable the kv.rangefeed.enabled :
If you are working on a CockroachDB Standard or Basic cluster, the kv.rangefeed.enabled cluster setting is enabled by default.
  1. Create a database called cdc_demo:
  2. Set the database as the default:
  3. Create a table and add data:
  4. Create another table and add data:
  5. Start the changefeed:
    To connect to Confluent Cloud, use the following URL structure: 'https://{API_KEY_ID}:{API_SECRET_URL_ENCODED}@{CONFLUENT_REGISTRY_URL}:443'. See the tutorial for further detail.
    This will start up the changefeed in the background and return the job_id. The changefeed writes to Kafka.
  6. In a new terminal, move into the extracted confluent-<version directory and start watching the Kafka topics:
    The initial scan displays the state of the table as of when the changefeed started (therefore, the initial value of "Petee" is omitted).
This example only prints the value. To print both the key and value of each message in the changefeed (e.g., to observe what happens with DELETEs), use the --property print.key=true flag.
  1. Back in the SQL client, insert more data:
  2. Back in the terminal where you’re watching the Kafka topics, the following output has appeared:
  3. When you are done, exit the SQL shell (\q).
  4. To stop cockroach: Get the process ID of the node:
    Gracefully shut down the node, specifying its process ID:
  5. To stop Kafka, move into the extracted confluent-<version directory and stop Confluent:

​
Create a changefeed connected to a Confluent Cloud sink

In this example, you’ll set up a changefeed for a single-node cluster that is connected to a Confluent Cloud managed Kafka cluster. The changefeed will watch a table and send messages to the sink.
  1. Use the command to start a single-node cluster:
  2. In this example, you’ll run CockroachDB’s application workload to set up some data for your changefeed. In a new terminal, first create the schema for the workload:
    Then run the workload:
  3. As the root user, open the :
  4. Enable the kv.rangefeed.enabled :
If you are working on a CockroachDB Standard or Basic cluster, the kv.rangefeed.enabled cluster setting is enabled by default.
  1. To prepare the Confluent Cloud Kafka sink, sign up on the Confluent Cloud trial page.
  2. On the Create cluster page, select the necessary cluster type. The Basic cluster type is sufficient to run this example. Select the configuration and regions. After you have confirmed payment and set a cluster name, Launch the cluster.
  3. From the Overview page, click on API Keys in the navigation menu. Click Create key and select the scope for this API key. Global access is sufficient for this example. Granular access is more suitable for production. Copy or download the key and secret.
  4. Create a topic in Confluent Cloud. Under Topics select Add topic. Add a topic name and define the number of partitions and then Create. CockroachDB defaults to using the table name as the topic name. If you would like to send messages to an alternate topic, you can specify the parameter.
You can enable auto topic creation for Confluent Cloud Dedicated clusters under Cluster Settings in the console.
  1. Construct the URI to connect your changefeed to the Confluent Cloud cluster. You will need:
    • The prefix scheme confluent-cloud://.
    • The address and port of the bootstrap server. Click on Cluster settings in your Confluent Cloud console. Under Endpoints, you will find the Bootstrap server. It will be something like: pkc-lzvrd.us-west4.gcp.confluent.cloud:9092.
    • Ensure that you follow the cluster address with ? before the remaining parameters, and also connect the following parameters with &.
    • Your API key and secret from the previous step passed with the api_key and api_secret parameters. You must URL encode the secret before adding to the connection string.
    • (Optional) Any further parameters. Refer to .
  2. Back in the SQL shell, create a changefeed:

​
Create a changefeed connected to a Google Cloud Pub/Sub sink

In this example, you’ll set up a changefeed for a single-node cluster that is connected to a Google Cloud Pub/Sub sink. The changefeed will watch a table and send messages to the sink.You’ll need access to a Google Cloud Project to set up a Pub/Sub sink. In this example, the Google Cloud CLI (gcloud) is used, but you can also complete each of these steps within your Google Cloud Console.
  1. Use the command to start a single-node cluster:
  2. In this example, you’ll run CockroachDB’s application workload to set up some data for your changefeed. In a new terminal, first create the schema for the workload:
    Then run the workload:
  3. As the root user, open the :
  4. Enable the kv.rangefeed.enabled :
If you are working on a CockroachDB Standard or Basic cluster, the kv.rangefeed.enabled cluster setting is enabled by default.
  1. Next, you’ll prepare your Pub/Sub sink. In a new terminal window, create a Service Account attached to your Google Project:
    In this example, cdc-demo will represent the name of the service account, and cockroach-project is the name of the Google Project. To ensure that your Service Account has the correct permissions to publish to the sink, use the following command to give the Service Account the predefined Pub/Sub Editor role:
  2. Create the Pub/Sub to which your changefeed will emit messages:
    Run the following command to create a subscription within the movr-users topic:
  3. With the topic and subscription set up, you can now download your Service Account credentials. Use the gcloud iam service-accounts keys create command to specify where to download the JSON credential file (credentials.json):
    Next, base64 encode the file that contains the entire JSON credential object using the command specific to your platform. If you’re working on macOS:
    If you’re working on Linux, run the following to ensure that lines are not wrapped in the output:
    Copy the output so that you can add it to your statement in the next step. When you create your changefeed, it is necessary that the credentials are base64 encoded before passing it in the URI.
  4. Back in the SQL shell, create a changefeed that will emit messages to your Pub/Sub topic. Ensure that you have base64 encoded the entire credentials JSON object for your Service Account and then run:
    You can include the region parameter for your topic, or use the option for multi-region Pub/Sub. See the page for more detail. The output will confirm the topic where the changefeed will emit messages to.
    To view all the messages delivered to your topic, you can use:
    • The Google Cloud Console. From the Pub/Sub menu, select Subscriptions in the left-hand navigation and then select the subscription ID from your list of subscriptions. On the subscription’s overview, click Messages, and then Pull to view messages.
    • The gcloud CLI. From your terminal, run the following command:
      This command will only pull these messages once per subscription. For example, if you ran this command again you would receive 10 different messages in your output. To receive more than one message at a time, pass the --limit flag. For more details, refer to the gcloud pubsub subscriptions pull documentation.

​
Create a changefeed connected to a cloud storage sink

In this example, you’ll set up a changefeed for a single-node cluster that is connected to an AWS S3 sink. The changefeed watches two tables. Note that you can set up changefeeds for any of .
  1. Use the command to start a single-node cluster:
  2. As the root user, open the :
  3. Enable the kv.rangefeed.enabled :
If you are working on a CockroachDB Standard or Basic cluster, the kv.rangefeed.enabled cluster setting is enabled by default.
  1. Create a database called cdc_demo:
  2. Set the database as the default:
  3. Create a table and add data:
  4. Create another table and add data:
  5. Start the changefeed:
    This will start up the changefeed in the background and return the job_id. The changefeed writes to AWS.
  6. Monitor your changefeed on the DB Console. For more information, see .
  7. When you are done, exit the SQL shell (\q).
  8. To stop cockroach: Get the process ID of the node:
    Gracefully shut down the node, specifying its process ID:

​
Create a changefeed connected to an Azure Event Hubs sink

In this example, you’ll set up a changefeed for a single-node cluster that is connected to Azure Event Hubs. The changefeed watches two tables. You’ll need access to the Azure Developer Portal, you can sign up for an Azure free account.
  1. Use the command to start a single-node cluster:
  2. In this example, you’ll run CockroachDB’s application workload to set up some data for your changefeed. In a new terminal, first create the schema for the workload:
    Then run the workload:
  3. As the root user, open the :
  4. Enable the kv.rangefeed.enabled :
If you are working on a CockroachDB Standard or Basic cluster, the kv.rangefeed.enabled cluster setting is enabled by default.
  1. To prepare the Azure Event Hubs sink, in the Portal, create a resource group.
  2. Once you have a resource group, select Event Hubs in the portal navigation menu and then + Create from the Event Hubs page tool bar to create a namespace. Complete the naming details for the namespace and select the resource group you created in the previous step. For Pricing Tier, ensure that you select Standard or Premium (the Basic tier does not support Apache Kafka workloads). Complete your configuration options, and then create the namespace.
  3. Select Go to resource after the deployment is complete. On the namespace overview page, select + Event Hub in the tool bar to create an event hub within this namespace. An Event Hub is equivalent to a Kafka topic. On the creation page, add a name and the configuration for partitions and message retention.
You can manually create an Event Hub, or Event Hubs will create automatically when you start the changefeed.
  1. To send changefeed messages to your Event Hub, you will need a shared_access_key. Find Shared access policies in the left-hand navigation. Click + Add and provide a name for the policy as well as confirming the permissions. Once created, click on the policy name and then copy the Primary key — you will need this to create your changefeed.
  2. Build the connection string to your Event Hub. You will need the following:
    • The azure-event-hub:// or kafka:// URI scheme.
    • The Host name from your Event Hubs namespace overview page. It will be something like: {your-event-hubs-namespace}.servicebus.windows.net.
    • The port :9093 for the Kafka protocol.
    • The shared_access_key_name parameter with the name of your policy.
    • The shared_access_key parameter with your URL-encoded primary key.
You can also use a kafka:// scheme in the URI:
You can include the or parameters to create and name an Event Hub for emitted messages.
  1. Create an to store your connection string:
  2. Create the changefeed to your Event Hub:
  3. To confirm the messages emitted to your Event Hub, navigate to the Event Hub’s overview page that contains Requests, Messages, and Throughput counters. View the messages by clicking Process Data in the left-hand navigation menu. Then select the Enable real time insights from events feature box. On the Query page, your messages will load under Input Preview.

​
Create a changefeed connected to a webhook sink

In this example, you’ll set up a changefeed for a single-node cluster that is connected to a local HTTP server via a webhook. For this example, you’ll use an example HTTP server to test out the webhook sink.
  1. Use the command to start a single-node cluster:
  2. In this example, you’ll run CockroachDB’s application workload to set up some data for your changefeed. First create the schema for the workload:
    Then run the workload:
  3. As the root user, open the :
  4. Enable the kv.rangefeed.enabled :
If you are working on a CockroachDB Standard or Basic cluster, the kv.rangefeed.enabled cluster setting is enabled by default.
  1. In a separate terminal window, set up your HTTP server. Clone the test repository:
  2. Next make the script executable and then run the server (passing a specific port if preferred, otherwise it will default to :3000):
  3. Back in your SQL shell, run the following statement to create a changefeed that emits to your webhook sink:
    You set up a changefeed on the vehicles table, which emits changefeed messages to the local HTTP server. See the for more information on the options available for creating your changefeed to a webhook sink.
    In the terminal where your HTTP server is running, you’ll receive output similar to:
    For more detail on emitted changefeed messages, refer to the page.

​
Create a changefeed connected to an Apache Pulsar sink

This feature is in and subject to change. To share feedback and/or issues, contact Support.
In this example, you’ll set up a changefeed for a single-node cluster that is connected to an Apache Pulsar sink. The changefeed will watch a table and send messages to the sink.
  1. Use the command to start a single-node cluster:
  2. In this example, you’ll run CockroachDB’s application workload to set up some data for your changefeed. In a new terminal, first create the schema for the workload:
    Then run the workload:
  3. As the root user, open the :
  4. Enable the kv.rangefeed.enabled :
If you are working on a CockroachDB Standard or Basic cluster, the kv.rangefeed.enabled cluster setting is enabled by default.
  1. To prepare a Pulsar sink, refer to the Apache Pulsar documentation for setup guides to host Pulsar on a local cluster, Docker, or a Kubernetes cluster.
  2. In a terminal window where your Pulsar sink is hosted, start the cluster:
    If you’re running Pulsar in a Docker container, use the docker run command to start the cluster:
You can start a changefeed, and Pulsar will automatically use the table as the topic name.If you want to create a topic name first, use the pulsar-admin tool to specify the topic’s tenant and namespace. This example uses the default namespace:
For more detail on persistent topics and working with topic resources, refer to the Manage Topics Apache Pulsar documentation.
  1. Enter the SQL shell:
  2. Create your changefeed:
    By default, Apache Pulsar listens for client connections on port :6650. For more detail on configuration, refer to the Apache Pulsar documentation. Changefeeds emitting to a Pulsar sink do not support external connections or a number of changefeed options. For a full list, refer to the page.
  3. In a different terminal window, start a Pulsar consumer to read messages from the changefeed. This example consumes messages from the rides topic:
    If you’re running Pulsar in a Docker container, use the docker run command to start a consumer:
    You will receive the changefeed’s messages similar to the following:
    For more detail on emitted changefeed messages, refer to the page.