This is the how-to guide for streaming Postgres to GCP Pub/Sub. See the quickstart for a step-by-step walkthrough or the reference for details on all configuration options.
Prerequisites
If you’re self-hosting Sequin, you’ll need:- Sequin installed
- A database connected
- A Google Cloud project
- A Google Cloud project
Basic setup
Enable Pub/Sub
In the Google Cloud console, navigate to Pub/Sub and ensure it’s enabled for your project.Create a Pub/Sub topic
- Navigate to the Pub/Sub Topics page in your Google Cloud Console
- Click “CREATE TOPIC”
- Enter a topic ID (e.g., “postgres-changes”)
- (Optional) Enable “message retention” if you want to be able to replay messages
- Click “CREATE”
- Google Cloud project ID (e.g., “my-project-123”)
- Pub/Sub topic ID (e.g., “postgres-changes”)
Create a service account
- Navigate to the IAM & Admin > Service Accounts page
- Select a project (if one is not already selected)
- Click “CREATE SERVICE ACCOUNT”
- Enter a name (e.g., “sequin-publisher”)
- Click “CREATE AND CONTINUE”
- Add the following two roles:
- “Pub/Sub Publisher” (
roles/pubsub.publisher
) - “Pub/Sub Viewer” (
roles/pubsub.viewer
)
- “Pub/Sub Publisher” (
- Click “CONTINUE”
- Click “DONE”
- Click on the newly created service account
- Go to the “KEYS” tab
- Click “ADD KEY” > “Create new key”
- Choose “JSON” format
- Click “CREATE”
Create Pub/Sub sink
In Sequin, navigate to the “Sinks” tab, click “Create Sink”, and select “Pub/Sub Sink”.Configure the source
1
Select source tables
Under “Source”, select the schemas and tables you want to stream data from.
2
Add filters (optional)
Add filters to the sink to control which database changes are sent to your Pub/Sub topic.
3
Specify backfill
You can optionally indicate if you want Pub/Sub to receive a backfill of all or a portion of the table’s existing data. Backfills are useful if you want to use Pub/Sub to process historical data.You can backfill at any time. If you don’t want to backfill, toggle “Backfill” off.
4
Configure message grouping
Under “Message grouping”, you can control how messages are grouped for ordering:
- Enable message grouping (default): Messages are grouped by primary key to ensure ordered delivery within each group. This is recommended if you plan to use Pub/Sub subscriptions with “message ordering” enabled.
- Disable message grouping: Messages are not grouped, allowing for higher throughput but without ordering guarantees.
customer_id
).When message grouping is enabled and you use a Pub/Sub subscription with “message ordering” enabled, your system will receive events for each group in the same order they occurred in Postgres.Configure Pub/Sub
1
Enter Pub/Sub details
Fill in your Pub/Sub connection details:
- Project ID (required): Your Google Cloud project ID (e.g., “my-project-123”)
- You can find this on the home page of the Google Cloud Console
- Topic ID (required): The Pub/Sub topic ID (e.g., “postgres-changes”)
- Service Account Credentials (required): Paste the entire contents of your service account JSON key file
2
Confirm connection
Click “Test Connection” to verify that Sequin can connect to your Pub/Sub topic.
3
Create the sink
Give your sink a name, then click “Create Pub/Sub Sink”.
Verify & debug
To verify that your Pub/Sub sink is working:- Make some changes in your source table
- Verify that the count of messages for your sink increases in the Sequin web console
- In the Google Cloud Console:
- Navigate to your Pub/Sub topic
- Click on the “MESSAGES” tab
- Under “Step 2”, select a Cloud Pub/Sub subscription
- Click “PULL” to see messages in the topic
- Click the “Messages” tab to view the state of messages for your sink
- Click any failed message
- Check the delivery logs for error details, including any Pub/Sub API errors
- Service account credentials are valid and not expired
- Service account has the “Pub/Sub Publisher” role
- Project ID and Topic ID match exactly
- Topic exists in the specified project
Next steps
- Setup a processor Now that your Postgres data is flowing into Pub/Sub, you can setup a subscriber to read from the topic and process the data. See the Pub/Sub documentation for details on consuming messages. Refer to the Pub/Sub sink reference for the shape of messages that Sequin will publish.
- Deploy your implementation When you’re ready to deploy your implementation, see “How to deploy to production”.
- Advanced configuration For more about how Pub/Sub sinks work, see the Pub/Sub sink reference.