How to stream Postgres to AWS Kinesis
Connect Postgres to Kinesis with Sequin. Stream database changes for real-time analytics and event-driven pipelines.
This guide shows you how to set up Postgres change data capture (CDC) and stream changes to Kinesis using Sequin.
With Postgres data streaming to Kinesis, you can trigger workflows, feed analytics pipelines, and fan-out events to other AWS services.
By the end of this how-to, you’ll have database changes flowing to a Kinesis data stream.
Prerequisites
If you’re self-hosting Sequin, you’ll need:
If you’re using Sequin Cloud, you’ll need:
Basic setup
Create a Kinesis stream
If you don’t have a stream already, create one using the AWS console or the AWS cli:
Create an IAM user for Sequin
Create an IAM user with permission to put records into your stream:
Replace <your-stream-arn>
with your stream’s ARN.
Save the Access Key ID and Secret Access Key; you’ll use these when configuring the sink.
The kinesis:DescribeStream
permission is technically optional: it’s only used for testing the connection to Kinesis in the Sequin console.
Create Kinesis sink
Navigate to the “Sinks” tab, click “Create Sink”, and select “Kinesis Sink”.
Configure the source
Select source table
Under “Source”, select the table you want to stream data from.
Specify filters
For changes you can restrict to insert
, update
, and/or delete
actions. You can also apply filters.
Specify backfill
Optionally send a backfill of existing rows. Toggle “Backfill” off if you only need new changes.
Configure Kinesis
Enter stream details
Fill in the stream ARN as well as the AWS access key and secret you created for Sequin.
Create the sink
Give your sink a name, then click “Create Kinesis Sink”.
Verify & debug
To verify that your Kinesis sink is working:
- Make some changes in your source table.
- Verify that the count of messages for your sink increases in the Sequin web console.
- Use your preferred Kinesis consumer to read from the stream and confirm the events arrive.
If messages don’t seem to be flowing:
- Click the “Messages” tab to view the state of messages for your sink
- Click any failed message
- Check the delivery logs for error details, including any AWS API errors
Next steps
-
Process the events
Build a consumer to read from your Kinesis stream and process the data. Sequin’s messages will be base64-encoded JSON; refer to the Kinesis sink reference for more information.
-
Deploy your implementation
When you’re ready to deploy, see “How to deploy to production”.
-
Advanced configuration
For more about how Kinesis sinks work, see the Kinesis sink reference.
-
Start from scratch
If you need more detailed steps to get Kinesis up and running, see the Kinesis sink quickstart.