This guide shows you how to set up Postgres change data capture (CDC) and stream changes to Google Cloud Pub/Sub using Sequin.

With Postgres data streaming to Pub/Sub, you can trigger workflows, keep services in sync, build audit logs, maintain caches, and more.

By the end of this how-to, you’ll have database changes flowing to a Pub/Sub topic.

​
Prerequisites

If you’re self-hosting Sequin, you’ll need:

  1. Sequin installed
  2. A database connected
  3. A Google Cloud project

If you’re using Sequin Cloud, you’ll need:

  1. A Sequin Cloud account
  2. A database connected
  3. A Google Cloud project

​
Basic setup

​
Enable Pub/Sub

In the Google Cloud console, navigate to Pub/Sub and ensure it’s enabled for your project.

​
Create a Pub/Sub topic

  1. Navigate to the Pub/Sub Topics page in your Google Cloud Console
  2. Click “CREATE TOPIC”
  3. Enter a topic ID (e.g., “postgres-changes”)
  4. (Optional) Enable “message retention” if you want to be able to replay messages
  5. Click “CREATE”

Make note of your:

  • Google Cloud project ID (e.g., “my-project-123”)
  • Pub/Sub topic ID (e.g., “postgres-changes”)

​
Create a service account

  1. Navigate to the IAM & Admin > Service Accounts page
  2. Select a project (if one is not already selected)
  3. Click “CREATE SERVICE ACCOUNT”
  4. Enter a name (e.g., “sequin-publisher”)
  5. Click “CREATE AND CONTINUE”
  6. Add the following two roles:
    • “Pub/Sub Publisher” (roles/pubsub.publisher)
    • “Pub/Sub Viewer” (roles/pubsub.viewer)
  7. Click “CONTINUE”
  8. Click “DONE”
  9. Click on the newly created service account
  10. Go to the “KEYS” tab
  11. Click “ADD KEY” > “Create new key”
  12. Choose “JSON” format
  13. Click “CREATE”

The JSON key file will be downloaded to your computer. Keep this safe - you’ll need it when configuring the Pub/Sub sink in Sequin.

​
Create Pub/Sub sink

In Sequin, navigate to the “Sinks” tab, click “Create Sink”, and select “Pub/Sub Sink”.

​
Configure the source

1

Select source tables

Under “Source”, select the schemas and tables you want to stream data from.

2

Add filters (optional)

Add filters to the sink to control which database changes are sent to your Pub/Sub topic.

3

Specify backfill

You can optionally indicate if you want Pub/Sub to receive a backfill of all or a portion of the table’s existing data. Backfills are useful if you want to use Pub/Sub to process historical data.

You can backfill at any time. If you don’t want to backfill, toggle “Backfill” off.

4

Specify message grouping

Under “Message grouping”, you’ll most likely want to leave the default option selected to ensure events for the same row are sent to Pub/Sub in order.

That way, if you’re using a Pub/Sub subscription with “message ordering” enabled, your system will receive events for a row in the same order they occurred in Postgres.

​
Configure Pub/Sub

1

Enter Pub/Sub details

Fill in your Pub/Sub connection details:

  • Project ID (required): Your Google Cloud project ID (e.g., “my-project-123”)
    • You can find this on the home page of the Google Cloud Console
  • Topic ID (required): The Pub/Sub topic ID (e.g., “postgres-changes”)
  • Service Account Credentials (required): Paste the entire contents of your service account JSON key file
2

Confirm connection

Click “Test Connection” to verify that Sequin can connect to your Pub/Sub topic.

3

Create the sink

Give your sink a name, then click “Create Pub/Sub Sink”.

​
Verify & debug

To verify that your Pub/Sub sink is working:

  1. Make some changes in your source table
  2. Verify that the count of messages for your sink increases in the Sequin web console
  3. In the Google Cloud Console:
    • Navigate to your Pub/Sub topic
    • Click on the “MESSAGES” tab
    • Under “Step 2”, select a Cloud Pub/Sub subscription
    • Click “PULL” to see messages in the topic

If messages don’t seem to be flowing:

  1. Click the “Messages” tab to view the state of messages for your sink
  2. Click any failed message
  3. Check the delivery logs for error details, including any Pub/Sub API errors

Common issues to check:

  • Service account credentials are valid and not expired
  • Service account has the “Pub/Sub Publisher” role
  • Project ID and Topic ID match exactly
  • Topic exists in the specified project

​
Next steps

  • Setup a processor

    Now that your Postgres data is flowing into Pub/Sub, you can setup a subscriber to read from the topic and process the data. See the Pub/Sub documentation for details on consuming messages.

    Refer to the Pub/Sub sink reference for the shape of messages that Sequin will publish.

  • Deploy your implementation

    When you’re ready to deploy your implementation, see “How to deploy to production”.

  • Advanced configuration

    For more about how Pub/Sub sinks work, see the Pub/Sub sink reference.