The Azure Event Hubs sink sends messages to an Azure Event Hub.

This is the reference for the Azure Event Hubs sink. See the quickstart for a step-by-step walkthrough or the how-to guide for an explanation of how to use the Azure Event Hubs sink.

​
Configuration

  • Namespace

    Your Event Hubs namespace (e.g., my-namespace).

  • Event Hub Name

    The name of your Event Hub (e.g., postgres-changes).

  • Shared Access Key Name

    The name of your Shared Access Policy (e.g., sequin-publisher).

  • Shared Access Key

    The primary or secondary key from your Shared Access Policy.

​
Service account permissions

Sequin requires a Shared Access Policy with the following permission:

  • Send: Allows publishing messages to the Event Hub

While you can use the default RootManageSharedAccessKey policy, it’s recommended to create a dedicated policy for Sequin with only the Send permission.

​
Message format

Sequin sends messages to Event Hubs as JSON. You can find the shape of the messages in the messages reference.

​
Retry behavior

If Sequin is unable to deliver a message to Event Hubs, it will retry the message indefinitely. Sequin will exponentially back off the retry interval, with a maximum backoff of roughly 3 minutes.

​
Message size limits

Event Hubs has a maximum message size of 1MB. If Event Hubs rejects a message due to size limitations, Sequin will cancel the message’s delivery.

If you want to see logging/alerting for this situation, please upvote the corresponding issue.

​
Message ordering and partitioning

Event Hubs ensures message ordering within individual partitions. When messages are sent to the same partition, they are guaranteed to be delivered in the order they were published.

When you configure an Event Hubs sink in Sequin, you can specify message grouping behavior. This will be used to determine how to order messages to send to Event Hubs.

The default message group for a message is the source row’s primary key(s). You can override this by specifying one or more columns to use for message grouping.

Sequin will order the delivery of messages with the same group according to their commit timestamp.

However, Sequin does not set the partition key on each message. That means that if you’re using multiple partitions in Event Hubs, messages with the same group can be sent to different partitions.

If you want to see support for setting the partition key on each message, please comment on the corresponding issue.

​
Routing

The Azure Event Hubs sink supports dynamic routing of the event_hub_name with routing functions.

Example routing function:

def route(action, record, changes, metadata) do
  %{
    event_hub_name: "#{metadata.table_schema}-#{metadata.table_name}"
  }
end

When not using a routing function, messages will be published to the Event Hub specified in the sink configuration.

​
Debugging

You can view the status of your Event Hubs sink in the Sequin web console.

On the “Messages” tab, you can see which messages are in-flight to Event Hubs, which messages Sequin is unable to deliver, and recently delivered messages.

Messages that Sequin is unable to deliver will have a “Deliver count” greater than 1. You can click on a message to see more details, including the last error response received from Event Hubs.

Common issues to check:

  • Shared Access Key is valid and not expired
  • Shared Access Policy has Send permission
  • Namespace and Event Hub name match exactly
  • Event Hub exists in the specified namespace