How to stream Postgres to a webhook endpoint
Build event-driven applications with Postgres change data capture (CDC) and webhooks. Learn to create reliable HTTP integrations with automatic retries.
This guide shows you how to set up Postgres change data capture (CDC) and stream changes to a webhook endpoint using Sequin.
With Postgres data streaming to your application, you can trigger workflows, keep services in sync, build audit logs, maintain caches, and more.
By the end of this how-to, you’ll have a configured HTTP endpoint receiving database changes.
This is the how-to guide for streaming Postgres to a webhook endpoint. See the quickstart for a step-by-step walkthrough or the reference for details on all configuration options.
Prerequisites
If you’re self-hosting Sequin, you’ll need:
If you’re using Sequin Cloud, you’ll need:
Basic setup
Prepare your HTTP endpoint
Sequin converts Postgres changes and rows into JSON and sends them to your HTTP endpoint.
You’ll need to create an HTTP endpoint on your server that Sequin can send data to. You can always start by creating a simple request handler, and then update it to process payloads after you get things working end-to-end.
To see the shape of the webhook payloads that Sequin sends, see the webhook sink reference.
Your webhook endpoint must return a 200 OK after successfully processing a batch of messages. Sequin will consider any other response code a failure, and will mark the message for re-delivery.
If you’re using Sequin Cloud and developing locally, you can use the Sequin CLI’s tunnel command to connect Sequin’s servers to a local HTTP endpoint. So, no need to deploy your application just yet!
If you’re not ready to add an HTTP endpoint to your application just yet, you can use a service like Webhook.site to receive webhooks.
Authentication
Before deploying your HTTP endpoint to production, you should add authentication. This way, your application can verify that requests are coming from Sequin.
A simple strategy is to generate a secure token and require it in the Authorization header of inbound requests. You’ll supply this token to Sequin by adding it as a header for your HTTP endpoint.
Here’s how to generate a secure token with openssl:
(Optional) Enable retention for changes
Sequin stores insert, update, and delete changes in a buffer until they’re delivered to your webhook endpoint.
You can use change retention to persist changes to a table in your database. Then, you can stream that table to your webhook endpoint. This gives you the power to run backfills/replays of recent changes at any times.
Create HTTP endpoint
Under “Resources”, navigate to the “HTTP Endpoints” tab and click “Create HTTP Endpoint”.
When setting up a webhook sink, you can specify the request path as well as additional headers. So:
- For the “URL” field, fill in the base URL for your HTTP endpoint.
- For the “Headers” field, include essential headers for your HTTP endpoint (e.g.
Content-Type: application/json).
Add sensitive headers like Authorization under “Encrypted Headers”.
Create webhook sink
Navigate to the “Sinks” tab, click “Create Sink”, and select “Webhook Sink”.
Configure the source
Select source table or schema
Under “Source”, select the schemas and tables you want to stream data from.
Add filters (optional)
Add filters to the sink to control which database changes are sent to your webhook endpoint.
Specify backfill
You can optionally indicate if you want your webhook endpoint to receive a backfill of all or a portion of the table’s existing data. Backfills are useful if you want your webhook endpoint to process historical data. For example, if you’re materializing a cache, you might want to warm it with existing rows.
You can backfill at any time. If you don’t want to backfill, toggle “Backfill” off.
Specify message grouping
Under “Message grouping”, you’ll most likely want to leave the default option selected to ensure events for the same row are sent to your webhook endpoint in order.
While your webhook endpoint is processing a message, if a new change or version related to the same row is captured, Sequin will hold the message back until your webhook endpoint acknowledges the first message. This is almost always the desired behavior.
Configure delivery
Specify request timeout
Under “Delivery configuration”, choose a conservative value for “Request timeout”. The right value depends on how long you want to give your webhook endpoint to process a batch of messages. When the request timeout is reached, Sequin will mark the message for re-delivery.
If your programming platform allows it, we recommend wrapping the webhook request handler in your application with a hard timeout limit below the “Request timeout” specified here. That way, if your handler is taking too long, it will be killed before Sequin retries the message.
Specify batch size
The right value for “Batch size” depends on your requirements.
You should try to process the full batch atomically.
For upserting to a database or store, set this to the number of rows you want to upsert in a single transaction.
For side effects like sending an email, you should probably set this to 1. Otherwise, you risk your worker partially processing a batch then crashing. Then, when Sequin retries the batch, your handler will re-process the beginning of the batch.
Specify advanced delivery configuration
See the webhook sink reference for more information on advanced delivery configuration options.
Select the HTTP endpoint you created
Under “HTTP Endpoint”, select the HTTP endpoint you created earlier.
Create the sink
Give your sink a name, then click “Create Webhook Sink”.
Verify & debug
To verify that your webhook sink is working:
- Make some changes in your source table.
- Verify that the count of messages for your sink increases in the Sequin web console.
- Verify receipt of messages in your webhook endpoint logs.
If messages don’t seem to be flowing:
- Click the “Messages” tab to view the state of messages for your sink.
- Click any failed message.
- Check the delivery logs for error details, including the response from your webhook endpoint.
Next steps
Assuming you’ve followed the steps above for your local environment, “How to deploy to production” will show you how to deploy your implementation to production.
For advanced configuration and more about how webhooks work, see the webhook sink reference.