The Kafka sink sends messages to a Kafka topic.
This is the reference for the Kafka sink. See the quickstart for a step-by-step walkthrough or the how-to guide for an explanation of how to use the Kafka sink.

​
Configuration

  • Hosts A comma-separated list of host:port pairs for your Kafka brokers (e.g., localhost:9092,localhost:9093).
  • Topic The name of the Kafka topic where messages will be sent. Maximum length is 255 characters.
  • Username (optional) Username for SASL authentication.
  • Password (optional) Password for SASL authentication.
  • SASL Mechanism (optional) The SASL mechanism to use for authentication. Can be PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512. This is required if you specify a username and password.
  • TLS Enable TLS encryption for connections to Kafka. Currently, Sequin only supports TLS with verify none. If you need a different TLS configuration, please let us know.

​
Advanced configuration

  • Max Ack Pending Sets the maximum number of messages that can be in-flight at any time. This helps control message flow and prevents overwhelming the broker.

​
Message format

Sequin sends messages to Kafka as JSON. You can find the shape of the messages in the messages reference.

​
Retry behavior

If Sequin is unable to deliver a message to Kafka, it will retry the message indefinitely. Sequin will exponentially back off the retry interval, with a maximum backoff of roughly 3 minutes.

​
Message size limits

The maximum message size in Kafka is configured at the broker level. The default maximum message size in Kafka is 1MB. If you need to handle larger messages, you’ll need to adjust the message.max.bytes setting on your Kafka brokers. If Kafka rejects a message due to size limitations, Sequin will cancel the message’s delivery.
If you want to see logging/alerting for this situation, please upvote the corresponding issue.

​
Message ordering and partitioning

Kafka ensures message ordering within individual partitions. When messages are sent to the same partition, they are guaranteed to be delivered in the order they were produced. Partitioning plays a critical role in maintaining this order. Sequin leverages Kafka’s partitioning model by using the hash of a record’s primary key(s) as the default partition key. This strategy ensures that all messages related to the same record are routed to the same partition, preserving the order of updates for that record. You can override the default partition key by setting the message_key field in your routing function.

​
Security

Sequin supports the following security features for Kafka connections:
  • TLS encryption
  • SASL authentication, with support for PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512
Make sure your Kafka brokers are properly configured to accept the security settings you enable in Sequin.

​
Debugging

You can view the status of your Kafka sink in the Sequin web console. On the “Messages” tab, you can see which messages are in-flight to Kafka, which messages Sequin is unable to deliver, and recently delivered messages. Messages that Sequin is unable to deliver will have a “Deliver count” greater than 1. You can click on a message to see more details, including the last error response received from Kafka.