Webhook sink

The webhook sink sends messages to a webhook endpoint. This can be a webhook endpoint for one of your own services or for a third-party service.

​

Configuration

• HTTP endpoint

  The HTTP endpoint to send messages to.

• HTTP endpoint path

  The path to append to the base URL for this sink’s requests.

• Request timeout

  The maximum duration allowed for the HTTP request to complete. If the request doesn’t finish within this time, it will be considered failed and may be retried.

  You should set this to a conservative value, given how long you estimate your webhook handler will take to process messages.

  If your programming platform allows it, we recommend wrapping the webhook request handler in your application with a hard timeout limit below the “Request timeout” specified here. That way, if your handler is taking too long, it will be killed before Sequin retries the message.

• Batch size

  The maximum number of messages to include in a single HTTP request. Default is 1.

  When batching is enabled, multiple messages will be included in the request body’s data array. Your webhook handler should process all messages in the batch atomically.

  For side effects like sending emails, you should probably set this to 1. Otherwise, you risk your handler partially processing a batch then crashing. Then, when Sequin retries the batch, your handler will re-process the beginning of the batch.

  For upserting to a database or store, set this to the number of rows you want to upsert in a single transaction.

  Batching can increase throughput, but increases the payload size of each request.

• Max ack pending (advanced)

  The maximum number of messages allowed in a pending state (delivered but not yet acknowledged) for this sink. Note this includes failed messages that Sequin is backing off and will retry.

  “Max ack pending” creates back pressure to avoid overloading your webhook endpoint.

  If you’re unsure, leave this set to the default value.

​

Request format

The webhook sink sends HTTP POST requests with JSON bodies. The body contains an array of messages wrapped in a data field.

The format of messages depends on whether you’ve configured the sink for change messages or row messages:

​

Change Messages

{
  data: Array<{
    record: Record<string, any>;  // The row data
    changes: Record<string, any> | null;  // Previous values for changed fields
    action: "insert" | "update" | "delete" | "read";
    metadata: {
      table_schema: string;
      table_name: string;
      commit_timestamp: string;  // ISO timestamp
      consumer: {
        id: string;
        name: string;
      };
    };
  }>;
}

Example:

{
  "data": [
    {
      "record": {
        "id": 1234,
        "customer_id": 789,
        "status": "shipped"
      },
      "changes": {
        "status": "pending"
      },
      "action": "update",
      "metadata": {
        "table_schema": "public",
        "table_name": "orders",
        "commit_timestamp": "2024-03-20T15:30:00Z",
        "consumer": {
          "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
          "name": "orders_webhook"
        }
      }
    }
  ]
}

​

Row Messages

{
  data: Array<{
    record: Record<string, any>;  // The row data
    metadata: {
      table_schema: string;
      table_name: string;
      consumer: {
        id: string;
        name: string;
      };
    };
  }>;
}

Example:

{
  "data": [
    {
      "record": {
        "id": 1234,
        "customer_id": 789,
        "status": "shipped"
      },
      "metadata": {
        "table_schema": "public",
        "table_name": "orders",
        "consumer": {
          "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
          "name": "orders_webhook"
        }
      }
    }
  ]
}

​

Response format

Your webhook endpoint must return a 2XX OK response after successfully processing a batch of messages. Sequin will consider any response code outside the range of 200 to 299 a failure, and will mark the message for re-delivery.

​

Authentication

For production, we recommend you use encrypted headers to authenticate requests. Sequin will include these header(s) in its requests to the endpoint. You can then verify that the requests are coming from Sequin.

You can add encrypted headers to the HTTP endpoint.

A simple strategy is to generate a secure token and have Sequin include it as an Authorization bearer token in its requests. To generate a secure token with openssl, run:

openssl rand -base64 32

​

Retry behavior

Sequin considers a message delivered if it:

• Receives a 200 OK response from the webhook endpoint.
• Receives that response within the “Request timeout” specified for the sink.

If Sequin is unable to deliver a message, it will retry the message indefinitely. Sequin will exponentially back off the retry interval, with a maximum backoff of roughly 3 minutes.

​

Grouping and ordering

When you configure a webhook sink, you can specify message grouping behavior.

The default message group for a message is the source row’s primary key(s). You can override this by specifying one or more columns to use for message grouping.

Sequin will only deliver messages that have no other messages with the same group already in-flight, i.e. currently being delivered to your webhook endpoint and awaiting a response. This gives you serial processing of messages with the same group, which can eliminate a lot of race conditions.

What’s more, Sequin will order the delivery of messages with the same group according to their commit timestamp. This means that for change messages, you can rely on receiving them in the order they occurred.

​

Batch behavior

When you specify a “Batch size” greater than 1, Sequin may include more than one message for the same group in a single request. For clarity, Sequin will never include more than the specified “Batch size” messages in a single request, but may include fewer.

​

Debugging

You can view the status of your webhook sink in the Sequin web console.

On the “Messages” tab, you can see which messages are in-flight, which messages Sequin is unable to deliver, and recently delivered messages.

Messages that Sequin is unable to deliver will have a “Deliver count” greater than 1. You can click on a message to see more details, including the last error response received from your endpoint.