Webhook sink

The webhook sink sends messages to a webhook endpoint. This can be a webhook endpoint for one of your own services or for a third-party service.

​

Configuration

• HTTP endpoint

  The HTTP endpoint to send messages to.

• HTTP endpoint path

  The path to append to the base URL for this sink’s requests.

• Request timeout

  The maximum duration allowed for the HTTP request to complete. If the request doesn’t finish within this time, it will be considered failed and may be retried.

  You should set this to a conservative value, given how long you estimate your webhook handler will take to process messages.

  If your programming platform allows it, we recommend wrapping the webhook request handler in your application with a hard timeout limit below the “Request timeout” specified here. That way, if your handler is taking too long, it will be killed before Sequin retries the message.

• Batch messages

  Whether to batch multiple messages into a single HTTP request. Defaults to false.

  When false, each HTTP request will contain a single message object as its body. You can use a transform to customize the shape of this object.

  When true, messages will be batched according to the “Batch size” and “Batch timeout” settings, and the request body will be a JSON object containing one or more messages.

• Batch size

  Only applicable when “Batch messages” is enabled.

  The maximum number of messages to include in a single HTTP request. Default is 1.

  When batching is enabled, multiple messages will be included in the request body’s data array. Your webhook handler should process all messages in the batch atomically.

  For side effects like sending emails, you should probably set this to 1. Otherwise, you risk your handler partially processing a batch then crashing. Then, when Sequin retries the batch, your handler will re-process the beginning of the batch.

  For upserting to a database or store, set this to the number of rows you want to upsert in a single transaction.

  Batching can increase throughput, but increases the payload size of each request.

• Batch timeout (advanced)

  Only applicable when “Batch messages” is enabled.

  The maximum time (in milliseconds) to wait for a batch to reach its full size before sending. Defaults to 50ms.

• Max ack pending (advanced)

  The maximum number of messages allowed in a pending state (delivered but not yet acknowledged) for this sink. Note this includes failed messages that Sequin is backing off and will retry.

  “Max ack pending” creates back pressure to avoid overloading your webhook endpoint.

  If you’re unsure, leave this set to the default value.

​

Request format

The webhook sink sends HTTP POST requests with JSON bodies.

• When “Batch messages” is disabled (default): The request body will contain a single message with the default payload shape. You can also use a transform to customize the payload.

• When “Batch messages” is enabled: The request body will contain a JSON object with a data field, which is an array of messages.

​

Message

{
  record: Record<string, any>;  // The row data
  changes: Record<string, any> | null;  // Previous values for changed fields
  action: "insert" | "update" | "delete" | "read";
  metadata: {
    table_schema: string;
    table_name: string;
    commit_timestamp: string;  // ISO timestamp
    commit_lsn: number;
    commit_idx: number;
    database_name: string; // deprecated
    consumer: {
      id: string;
      name: string;
      annotations: Record<string, any>;
    };
    database: {
      id: string;
      name: string;
      annotations: Record<string, any>;
      database: string;
      hostname: string;
    };
  };
}

Example:

{
  "record": {
    "id": 1234,
    "customer_id": 789,
    "status": "shipped"
  },
  "changes": {
    "status": "pending"
  },
  "action": "update",
  "metadata": {
    "table_schema": "public",
    "table_name": "orders",
    "commit_timestamp": "2024-03-20T15:30:00Z",
    "commit_lsn": 123456789,
    "commit_idx": 1,
    "database_name": "myapp-prod", // deprecated
    "consumer": {
      "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
      "name": "orders_webhook",
      "annotations": {"my-custom-key": "my-custom-value"}
    },
    "database": {
      "id": "12345678-9abc-def0-1234-56789abcdef0",
      "name": "myapp-prod",
      "annotations": {"my-custom-key": "my-custom-value"},
      "database": "myapp-prod",
      "hostname": "db.example.com"
    }
  }
}

Example (when batching is enabled):

{
  "data": [
    {
      "record": {
        "id": 1234,
        "customer_id": 789,
        "status": "shipped"
      },
      "changes": {
        "status": "pending"
      },
      "action": "update",
      "metadata": {
        "table_schema": "public",
        "table_name": "orders",
        "commit_timestamp": "2024-03-20T15:30:00Z",
        "commit_lsn": 123456789,
        "commit_idx": 1,
        "database_name": "myapp-prod", // deprecated
        "consumer": {
          "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
          "name": "orders_webhook",
          "annotations": {"my-custom-key": "my-custom-value"}
        },
        "database": {
          "id": "12345678-9abc-def0-1234-56789abcdef0",
          "name": "myapp-prod",
          "annotations": {"my-custom-key": "my-custom-value"},
          "database": "myapp-prod",
          "hostname": "db.example.com"
        }
      }
    }
  ]
}

​

Response format

Your webhook endpoint must return a 2XX OK response after successfully processing the received message or batch of messages. Sequin will consider any response code outside the range of 200 to 299 a failure, and will mark the message(s) for re-delivery.

​

Authentication

For production, we recommend you use encrypted headers to authenticate requests. Sequin will include these header(s) in its requests to the endpoint. You can then verify that the requests are coming from Sequin.

You can add encrypted headers to the HTTP endpoint.

A simple strategy is to generate a secure token and have Sequin include it as an Authorization bearer token in its requests. To generate a secure token with openssl, run:

openssl rand -base64 32

​

Retry behavior

Sequin considers a message delivered if it:

• Receives a 200 OK response from the webhook endpoint.
• Receives that response within the “Request timeout” specified for the sink.

If Sequin is unable to deliver a message, it will retry the message indefinitely. Sequin will exponentially back off the retry interval, with a maximum backoff of roughly 3 minutes.

​

Grouping and ordering

When you configure a webhook sink, you can specify message grouping behavior.

The default message group for a message is the source row’s primary key(s). You can override this by specifying one or more columns to use for message grouping.

Sequin will only deliver messages that have no other messages with the same group already in-flight, i.e. currently being delivered to your webhook endpoint and awaiting a response. This gives you serial processing of messages with the same group, which can eliminate a lot of race conditions.

What’s more, Sequin will order the delivery of messages with the same group according to their commit timestamp. This means that for change messages, you can rely on receiving them in the order they occurred.

​

Batch behavior

When “Batch messages” is enabled and you specify a “Batch size” greater than 1, Sequin may include more than one message for the same group in a single request. For clarity, Sequin will never include more than the specified “Batch size” messages in a single request, but may include fewer based on the “Batch timeout”.

​

Debugging

You can view the status of your webhook sink in the Sequin web console.

On the “Messages” tab, you can see which messages are in-flight, which messages Sequin is unable to deliver, and recently delivered messages.

Messages that Sequin is unable to deliver will have a “Deliver count” greater than 1. You can click on a message to see more details, including the last error response received from your endpoint.