Get started with AWS Kinesis
Stream Postgres changes to an Amazon Kinesis stream in minutes with Sequin.
In this quickstart, you’ll create a real-time data pipeline that streams changes from a Postgres database to an Amazon Kinesis data stream. You’ll:
- Set up a Kinesis stream using the AWS CLI
- Create an IAM user with the minimal required permissions
- Boot Sequin and connect to a sample playground database
- Configure Sequin to stream database changes to your Kinesis stream
- See your changes flow in real-time
By the end, you’ll have hands-on experience setting up Postgres change data capture (CDC) with Sequin and Kinesis.
Prerequisites
- AWS CLI installed and configured with your AWS credentials
- An AWS account with permissions to create Kinesis streams and IAM users
Step 1: Set up your Kinesis stream
First, let’s create a Kinesis stream and the necessary AWS resources using the AWS CLI.
Create a Kinesis stream
Run this command to create a new Kinesis stream called sequin-quickstart
:
Wait for the stream to become active (this takes about 1-2 minutes):
Look for "StreamStatus": "ACTIVE"
in the response.
Get your stream ARN
Once active, note down your stream ARN from the describe-stream output. It will look like:
You can also get just the ARN with:
Create an IAM policy
Create a policy file called sequin-kinesis-policy.json
:
Create the policy in AWS:
Note the policy ARN from the response - you’ll need it in the next step.
Create an IAM user
Create a dedicated IAM user for Sequin:
Attach the policy to the user (replace YOUR_ACCOUNT_ID
with your AWS account ID):
If you don’t have your account ID handy, you can ask AWS like this:
Create access keys
Generate access keys for the user:
Important: Save the AccessKeyId
and SecretAccessKey
from the response - you’ll need these for Sequin configuration and they won’t be shown again.
Run Sequin
The easiest way to get started with Sequin is with our Docker Compose file. This file starts a Postgres database, Redis instance, and Sequin server.
Create directory and start services
- Download sequin-docker-compose.zip.
- Unzip the file.
- Navigate to the unzipped directory and start the services:
Verify services are running
Check that Sequin is running using docker ps
:
You should see output like the following:
Sequin, Postgres, Redis, Prometheus, and Grafana should be up and running (status: Up
).
Login
The Docker Compose file automatically configures Sequin with an admin user and a playground database.
Let’s log in to the Sequin web console:
Open the web console
After starting the Docker Compose services, open the Sequin web console at http://localhost:7376:
Login with default credentials
Use the following default credentials to login:
- Email:
- Password:
View the playground database
To get you started quickly, Sequin’s Docker Compose file creates a logical database called sequin_playground
with a sample dataset in the public.products
table.
Let’s take a look:
Navigate to Databases
In the Sequin web console, click Databases in the sidebar.
Select playground database
Click on the pre-configured sequin-playground
database:
The database “Health” should be green.
View contents of the products table
Let’s get a sense of what’s in the products
table. Run the following command:
This command connects to the running Postgres container and runs a psql
command.
You should see a list of the rows in the products
table:
We’ll make modifications to this table in a bit.
Create a Kinesis Sink
With the playground database connected, create a sink that will send changes to your Kinesis stream:
Navigate to Sinks
Click “Sinks” in the sidebar navigation, then click “Create Sink”.
Select sink type
Select “Amazon Kinesis” as the sink type and click “Continue”.
Note "Source" configuration
In the “Source” card, note that the sequin-playground
database and products
table are pre-selected. Leave these defaults:
Setup a backfill
Toggle open the “Initial backfill” card and select “Backfill all rows”:
Configure Kinesis connection
In the Kinesis configuration card, enter:
- Stream ARN: The ARN you noted from Step 1 (e.g.,
arn:aws:kinesis:us-east-1:123456789012:stream/sequin-quickstart
) - AWS Access Key ID: The
AccessKeyId
from Step 1 - AWS Secret Access Key: The
SecretAccessKey
from Step 1 - AWS Region: The region where you created your stream (e.g.,
us-east-1
)
Create the sink
Click “Create Sink” to finish setting up your Kinesis sink.
See changes flow to your Kinesis stream
On the new sink’s overview page, you should see the “Health” status turn green, indicating data is flowing to your stream.
Messages tab
Click the “Messages” tab to see recently delivered messages.
Verify in AWS (optional)
You can also verify data is flowing by checking your Kinesis stream in the AWS Console or using the CLI:
Great work!
You’ve successfully:
- Created a Kinesis stream with the AWS CLI
- Set up an IAM user with minimal required permissions
- Connected Sequin to stream Postgres changes to Kinesis
- Loaded existing data through a backfill
- Verified changes are flowing to your Kinesis stream
Clean up (optional)
If you want to clean up the AWS resources created in this quickstart:
Ready to stream
Now you’re ready to connect your own database to Sequin and start streaming changes: