How to stream Postgres to AWS SQS
Receive Postgres changes in AWS SQS in real-time
This guide shows you how to set up Postgres change data capture (CDC) and stream changes to SQS using Sequin.
With Postgres data streaming to SQS, you can trigger workflows, keep services in sync, build audit logs, maintain caches, and more.
By the end of this how-to, you’ll have database changes flowing to an SQS queue.
Prerequisites
If you’re self-hosting Sequin, you’ll need:
If you’re using Sequin Cloud, you’ll need:
Basic setup
Create an SQS queue
If you don’t have an SQS queue already, create one in your AWS account.
Most likely, you’ll want to create a FIFO queue. This will ensure your system receives events pertaining to the same row in order.
Create an IAM user for Sequin
First, create an IAM user that Sequin will use to send messages to your SQS queue:
- Navigate to the IAM service in your AWS Console
- Click “Users” then “Create user”
- Name the user (e.g., “sequin-sqs-publisher”)
- Select “Access key - Programmatic access”
Replace <your-queue-arn> with your actual queue ARN (e.g., arn:aws:sqs:us-east-1:123456789012:my-queue).
After creating the user, save the Access Key ID and Secret Access Key - you’ll need these when configuring the SQS sink in Sequin in a moment.
(Optional) Enable retention for changes
You can send either changes or rows to SQS.
If you’re sending changes, note that changes are ephemeral. Sequin stores insert, update, and delete changes in a buffer table until they’re delivered to SQS.
You can use change retention to persist changes to a table in your database. Then, you can stream that table to SQS. This gives you the power to run backfills/replays of recent changes at any times.
Create SQS sink
Navigate to the “Sinks” tab, click “Create Sink”, and select “SQS Sink”.
Configure the source
Select source table
Under “Source”, select the table you want to stream data from.
Choose message type
Specify whether you want to receive changes or rows from the table.
If you’re not sure which to choose, you can leave the default option “Changes” selected. Changes are a better fit for event-driven use cases, like triggering workflows, which is usually what you want to do with SQS.
Specify filters
If you selected changes, in “Records to process”, you can indicate whether you want to receive insert, update, and/or delete changes.
You can also specify SQL filters to narrow down the events you want to receive. For example, if you only want to receive events for subscriptions that currently have an mrr greater than $100, you can add a filter on mrr > 100.
Specify backfill
You can optionally indicate if you want SQS to receive a backfill of all or a portion of the table’s existing data. Backfills are useful if you want to use SQS to process historical data. For example, if you’re materializing a cache, you might want to warm it with existing rows.
You can backfill at any time. If you don’t want to backfill, toggle “Backfill” off.
Specify message grouping
Under “Message grouping”, you’ll most likely want to leave the default option selected to ensure events for the same row are sent to SQS in order.
That way, if you’re using an SQS FIFO queue, your system will receive events for a row in the same order they occurred in.
Configure SQS
Enter queue details
Fill in the queue’s URL, as well as the AWS client ID and secret you configured for Sequin.
If your queue is a FIFO queue, be sure to include the .fifo extension in the queue’s URL.
Create the sink
Give your sink a name, then click “Create SQS Sink”.
Verify & debug
To verify that your SQS sink is working:
- Make some changes in your source table.
- Verify that the count of messages for your sink increases in the Sequin web console.
- In the AWS Console, navigate to your queue. Click “Send and receive messages” then “Poll for messages”. You should see the messages from Sequin appear in the table below.
If messages don’t seem to be flowing:
- Click the “Messages” tab to view the state of messages for your sink
- Click any failed message
- Check the delivery logs for error details, including any AWS API errors
Next steps
-
Setup a processor
Now that your Postgres data is flowing into SQS, you can setup a processor to read from the queue and process the data. See the AWS docs on receiving and processing messages.
Refer to the SQS sink reference for the shape of messages that Sequin will publish to SQS.
-
Deploy your implementation
When you’re ready to deploy your implementation, see “How to deploy to production”.
-
Advanced configuration
For more about how SQS sinks work, see the SQS sink reference.