This guide shows you how to set up Postgres change data capture (CDC) and stream changes to a Sequin Stream.

With a Sequin Stream, you can pull database changes directly from Sequin without setting up additional infrastructure like Kafka or SQS. Sequin Stream provides exactly-once processing guarantees and supports multiple consumers working together as a consumer group.

By the end of this how-to, you’ll have database changes available to pull via HTTP.

​
Prerequisites

If you’re self-hosting Sequin, you’ll need:

  1. Sequin installed
  2. A database connected

If you’re using Sequin Cloud, you’ll need:

  1. A Sequin Cloud account
  2. A database connected

​
Basic setup

​
(Optional) Enable retention for changes

You can send either changes or rows to a Sequin Stream.

If you’re sending changes, note that changes are ephemeral. Sequin stores insert, update, and delete changes in a buffer table until they’re delivered to your consumer.

You can use change retention to persist changes to a table in your database. Then, you can stream that table to your consumer. This gives you the power to run backfills/replays of recent changes at any time. This can be handy: for example, if you realize there’s a bug in your message handling implementation, you can deploy the fix and re-process changes from the last few minutes or days.

​
Create Sequin Stream sink

Navigate to the “Sinks” tab, click “Create Sink”, and select “Sequin Stream Sink”.

​
Configure the source

1

Select source table

Under “Source”, select the table you want to stream data from.

2

Choose message type

Specify whether you want to receive changes or rows from the table.

If you’re not sure which to choose, you can start with the default, “Changes”.

3

Specify filters

If you selected changes, in “Records to process”, you can indicate whether you want to receive insert, update, and/or delete changes.

You can also specify SQL filters to narrow down the events you want to receive. For example, if you only want to receive events for subscriptions that currently have an mrr greater than $100, you can add a filter on mrr > 100.

4

Specify backfill

You can optionally indicate if you want to receive a backfill of all or a portion of the table’s existing data. Backfills are useful if you want to process historical data.

You can backfill at any time. If you don’t want to backfill, toggle “Backfill” off.

5

Specify message grouping

Under “Message grouping”, you’ll most likely want to leave the default option selected to ensure events for the same row are delivered in order.

While your consumer is processing a message, if a new change or version related to the same row is captured, Sequin will hold the message back until a consumer acknowledges the first message. This is almost always the desired behavior.

​
Configure delivery

1

Specify visibility timeout

Under “Delivery configuration”, choose a conservative value for “Visibility timeout”. This is how long a message will be invisible to other consumers after being received. When the timeout is reached, Sequin will make the message available again if it hasn’t been acknowledged.

The right value depends on how long you expect your consumer to take to process a batch of messages.

2

Specify max ack pending

Under “Max ack pending”, specify the maximum number of messages that can be outstanding (e.g. in-flight but not yet acknowledged by your consumer group) at one time.

This setting helps prevent issues where messages fail to be acknowledged, either due to errors in processing or bugs that cause too many workers to spin up. When the max ack pending limit is reached, Sequin stops delivering new messages and focuses on re-delivering unacknowledged messages.

The right value depends on your consumer group’s processing capacity.

3

Specify max waiting

Under “Max waiting” you can specify that maximum number of consumers that can be waiting to receive messages at one time.

This setting helps manage your consumer group’s resource utilization and prevent excessive polling of the Sequin Stream API. The right value depends on how many parallel consumers you expect to have in your consumer group.

4

Create the sink

Give your sink a name, then click “Create Sequin Stream Sink”.

​
Verify & test

To verify that your sink is working:

  1. Make some changes in your source table.

  2. Verify that the count of messages for your sink increases in the Sequin web console.

  3. Test receiving messages using curl. Find the receive curl request on your sink’s details page:

    curl -X GET "https://api.sequinstream.com/api/http_pull_consumers/{{YOUR_CONSUMER_NAME}}/receive?batch_size=10" \
  -H "Authorization: Bearer {{YOUR_TOKEN}}"

    You should see a response containing your messages:

    {
  "data": [
    {
      "ack_id": "0e6ae50d-4226-4498-8429-fc012737125b",
      "data": {
        "record": {
          "id": 1,
          "name": "Example Record",
          "created_at": "2024-03-20T15:30:00Z"
        }
      }
    }
  ]
}

  4. Acknowledge the messages you received:

    curl -X POST "https://api.sequinstream.com/api/http_pull_consumers/{{YOUR_CONSUMER_NAME}}/ack" \
  -H "Authorization: Bearer {{YOUR_TOKEN}}" \
  -d '{"ack_ids": ["0e6ae50d-4226-4498-8429-fc012737125b"]}'

    You should see the message state changes to acknowledged in the Sequin web console.

​
Next steps

  • Setup your consumer group to process messages

    Now that your Postgres data is flowing into Sequin Stream, you can setup a consumer group to read and process the data. See the Sequin Stream API reference for more information on receiving messages, acknowledging messages, and more.

  • Deploy your implementation

    When you’re ready to deploy your implementation, see “How to deploy to production”.

  • Advanced configuration

    For more about how Sequin Stream sinks work, see the Sequin Stream sink reference.

Was this page helpful?

RedisGCP Pub/Sub