This is the reference for the Sequin Stream sink. See the quickstart for a step-by-step walkthrough or the how-to guide for an explanation of how to use the Sequin Stream sink.
Configuration
When you create a Sequin Stream sink, you can configure the following settings:- Visibility timeout How long a message will be invisible to other consumers after being received. When the timeout is reached, Sequin will make the message available again if it hasn’t been acknowledged. Choose a conservative value based on how long you expect your consumer to take to process a batch of messages.
- Max ack pending The maximum number of messages that can be outstanding (i.e. in-flight but not yet acknowledged by your consumer group) at one time. This setting helps prevent issues where messages fail to be acknowledged, either due to errors in processing or bugs that cause too many workers to spin up. When the max ack pending limit is reached, Sequin stops delivering new messages and focuses on re-delivering unacknowledged messages.
-
Max waiting
The maximum number of consumers that can be waiting to receive messages at one time. Once reached, subsequent
/receive
calls will be rejected with a 429 error. This setting helps manage your consumer group’s resource utilization and prevent excessive polling of the Sequin Stream API. The right value depends on how many parallel consumers you expect to have in your consumer group.
We’re working to support multiple tables per Sequin Stream sink, but this is not yet available.
Consumer group pattern
The Sequin Stream API allows you to pull rows from your Sequin Streams using consumer groups. Consumer groups for a Sequin Stream are like consumer groups in other streaming systems. They provide a way to safely process data in parallel with exactly-once processing guarantees. In the consumer group pattern, each message (i.e. change or row from your database) goes to only one consumer in the group. If a consumer fails, other consumers can pick up its messages, ensuring nothing is missed. When you set up a Sequin Stream sink, Sequin provisions an HTTP endpoint for your consumer group to pull messages from using the Sequin Stream API:- Multiple consumers: You can have many processes or workers (i.e. consumers) pulling from the same endpoint simultaneously.
- Visibility timeout: After receiving a batch of messages, a consumer has a configurable period to process them before they become available to other consumers.
- Acknowledgment: Consumers explicitly acknowledge processed messages to ensure exactly-once processing.
- Automatic retries: If a consumer fails to process or acknowledge a batch, those messages become available for redelivery.
Processing messages
There are three steps in the lifecycle of processing messages with a Sequin Stream:Receive
Your consumer requests one or a batch of available messages from the consumer group by calling the
/receive
endpoint.Process
Your consumer processes the received messages.
Ack or Nack
After processing, your consumer sends an acknowledgement (
ack
) for successfully processed messages or a negative acknowledgement (nack
) for messages that couldn’t be processed.Receive messages
The first step is to make a call to the/receive
endpoint to get one or a batch of messages:
ack_id
and the default message payload:
Message
30
seconds and is configurable in your sink’s settings.
Acknowledge messages
Once your consumer has finished processing the messages, acknowledge orack
them. This tells Sequin you’re done processing them, and ensures that the consumer group won’t see them again:
Nack messages
Alternatively, if your consumer is unable to process the messages, it cannack
them. This tells Sequin to make the messages available for processing again right away:
nack
in the hopes that another consumer will pick up the messages that has a working connection.
Instead of nacking, your consumer can also do nothing. After the visibility timeout expires, the messages will be made available for processing again.