This guide shows you how to set up Postgres change data capture (CDC) and stream changes to Kinesis using Sequin.

With Postgres data streaming to Kinesis, you can trigger workflows, feed analytics pipelines, and fan-out events to other AWS services.

By the end of this how-to, you’ll have database changes flowing to a Kinesis data stream.

​
Prerequisites

If you’re self-hosting Sequin, you’ll need:

  1. Sequin installed
  2. A database connected

If you’re using Sequin Cloud, you’ll need:

  1. A Sequin Cloud account
  2. A database connected

​
Basic setup

​
Create a Kinesis stream

If you don’t have a stream already, create one using the AWS console or the AWS cli:

aws kinesis create-stream \
    --stream-name my-test-stream \
    --shard-count 1

​
Create an IAM user for Sequin

Create an IAM user with permission to put records into your stream:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kinesis:PutRecords", "kinesis:DescribeStream"],
      "Resource": "<your-stream-arn>"
    }
  ]
}

Replace <your-stream-arn> with your stream’s ARN.

Save the Access Key ID and Secret Access Key; you’ll use these when configuring the sink.

The kinesis:DescribeStream permission is technically optional: it’s only used for testing the connection to Kinesis in the Sequin console.

​
Create Kinesis sink

Navigate to the “Sinks” tab, click “Create Sink”, and select “Kinesis Sink”.

​
Configure the source

1

Select source table or schema

Under “Source”, select the table or schema you want to stream data from.

2

Specify filters

For changes you can restrict to insert, update, and/or delete actions. You can also apply filters.

3

Specify backfill

Optionally send a backfill of existing rows. Toggle “Backfill” off if you only need new changes.

​
Configure Kinesis

1

Enter stream details

Fill in the stream ARN as well as the AWS access key and secret you created for Sequin.

2

Create the sink

Give your sink a name, then click “Create Kinesis Sink”.

​
Verify & debug

To verify that your Kinesis sink is working:

  1. Make some changes in your source table.
  2. Verify that the count of messages for your sink increases in the Sequin web console.
  3. Use your preferred Kinesis consumer to read from the stream and confirm the events arrive.

If messages don’t seem to be flowing:

  1. Click the “Messages” tab to view the state of messages for your sink
  2. Click any failed message
  3. Check the delivery logs for error details, including any AWS API errors

​
Next steps

  • Process the events

    Build a consumer to read from your Kinesis stream and process the data. Sequin’s messages will be base64-encoded JSON; refer to the Kinesis sink reference for more information.

  • Deploy your implementation

    When you’re ready to deploy, see “How to deploy to production”.

  • Advanced configuration

    For more about how Kinesis sinks work, see the Kinesis sink reference.

  • Start from scratch

    If you need more detailed steps to get Kinesis up and running, see the Kinesis sink quickstart.