This guide shows you how to set up Postgres change data capture (CDC) and stream changes to a Sequin Stream.

With a Sequin Stream, you can pull database changes directly from Sequin without setting up additional infrastructure like Kafka or SQS. Sequin Stream provides exactly-once processing guarantees and supports multiple consumers working together as a consumer group.

By the end of this how-to, you’ll have database changes available to pull via HTTP.

This is the how-to guide for streaming Postgres to Sequin Stream. See the quickstart for a step-by-step walkthrough or the reference for details on all configuration options.

​
Prerequisites

If you’re self-hosting Sequin, you’ll need:

  1. Sequin installed
  2. A database connected

If you’re using Sequin Cloud, you’ll need:

  1. A Sequin Cloud account
  2. A database connected

​
Basic setup

​
(Optional) Enable retention for changes

Sequin stores insert, update, and delete changes in a buffer table until they’re delivered to your consumer.

You can use change retention to persist changes to a table in your database. Then, you can stream that table to your consumer. This gives you the power to run backfills/replays of recent changes at any time. This can be handy: for example, if you realize there’s a bug in your message handling implementation, you can deploy the fix and re-process changes from the last few minutes or days.

​
Create Sequin Stream sink

Navigate to the “Sinks” tab, click “Create Sink”, and select “Sequin Stream Sink”.

​
Configure the source

1

Select source tables

Under “Source”, select the schemas and tables you want to stream data from.

2

Add filters (optional)

Add filters to the sink to control which database changes are sent to your Sequin Stream.

3

Add transform (optional)

Add a transform to the sink to modify the payload before it’s sent to Sequin Stream.

4

Specify backfill

You can optionally indicate if you want to receive a backfill of all or a portion of the table’s existing data. Backfills are useful if you want to process historical data.

You can backfill at any time. If you don’t want to backfill, toggle “Backfill” off.

5

Specify message grouping

Under “Message grouping”, you’ll most likely want to leave the default option selected to ensure events for the same row are delivered in order.

While your consumer is processing a message, if a new change or version related to the same row is captured, Sequin will hold the message back until a consumer acknowledges the first message. This is almost always the desired behavior.

​
Configure delivery

1

Specify visibility timeout

Under “Delivery configuration”, choose a conservative value for “Visibility timeout”. This is how long a message will be invisible to other consumers after being received. When the timeout is reached, Sequin will make the message available again if it hasn’t been acknowledged.

The right value depends on how long you expect your consumer to take to process a batch of messages.

2

Specify max ack pending

Under “Max ack pending”, specify the maximum number of messages that can be outstanding (e.g. in-flight but not yet acknowledged by your consumer group) at one time.

This setting helps prevent issues where messages fail to be acknowledged, either due to errors in processing or bugs that cause too many workers to spin up. When the max ack pending limit is reached, Sequin stops delivering new messages and focuses on re-delivering unacknowledged messages.

The right value depends on your consumer group’s processing capacity.

3

Specify max waiting

Under “Max waiting” you can specify that maximum number of consumers that can be waiting to receive messages at one time.

This setting helps manage your consumer group’s resource utilization and prevent excessive polling of the Sequin Stream API. The right value depends on how many parallel consumers you expect to have in your consumer group.

4

Create the sink

Give your sink a name, then click “Create Sequin Stream Sink”.

​
Verify & test

To verify that your sink is working:

  1. Make some changes in your source table.

  2. Verify that the count of messages for your sink increases in the Sequin web console.

  3. Test receiving messages using curl. Find the receive curl request on your sink’s details page:

    curl -X GET "https://api.sequinstream.com/api/http_pull_consumers/{{YOUR_CONSUMER_NAME}}/receive?batch_size=10" \
  -H "Authorization: Bearer {{YOUR_TOKEN}}"

    You should see a response containing your messages:

    {
  "data": [
    {
      "ack_id": "0e6ae50d-4226-4498-8429-fc012737125b",
      "data": {
        "record": {
          "id": 1,
          "name": "Example Record",
          "created_at": "2024-03-20T15:30:00.127470Z"
        }
      }
    }
  ]
}

  4. Acknowledge the messages you received:

    curl -X POST "https://api.sequinstream.com/api/http_pull_consumers/{{YOUR_CONSUMER_NAME}}/ack" \
  -H "Authorization: Bearer {{YOUR_TOKEN}}" \
  -d '{"ack_ids": ["0e6ae50d-4226-4498-8429-fc012737125b"]}'

    You should see the message state changes to acknowledged in the Sequin web console.

​
Next steps

  • Setup your consumer group to process messages

    Now that your Postgres data is flowing into Sequin Stream, you can setup a consumer group to read and process the data. See the Sequin Stream API reference for more information on receiving messages, acknowledging messages, and more.

  • Deploy your implementation

    When you’re ready to deploy your implementation, see “How to deploy to production”.

  • Advanced configuration

    For more about how Sequin Stream sinks work, see the Sequin Stream sink reference.