AWS Kinesis sink

The Kinesis sink sends messages to an Amazon Kinesis data stream.

Kinesis is a durable, distributed, sharded queue that stores messages for up to 1 year, with ordering guaranteed within each shard based on partition keys.

Unlike SQS or SNS, Kinesis maintains message ordering and enables multiple consumers to process the same data stream independently.

​

Configuration

• Stream ARN

  The ARN of your Kinesis data stream. Must be in the format arn:aws:kinesis:<region>:<account-id>:stream/<stream-name>.

• Access Key ID

  The AWS access key ID with permissions to put records to the stream.

• Secret Access Key

  The AWS secret access key corresponding to the access key ID.

​

IAM user permissions

Sequin requires an IAM user with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kinesis:PutRecords", "kinesis:DescribeStream"],
      "Resource": "<your-stream-arn>"
    }
  ]
}

Replace <your-stream-arn> with your actual stream ARN.

​

Message format

Sequin sends messages to Kinesis as base64-encoded JSON. You can find the shape of the messages in the messages reference.

​

Retry behavior

If Sequin is unable to deliver a message to Kinesis, it will retry the message indefinitely. Sequin will exponentially back off the retry interval, with a maximum backoff of roughly 3 minutes.

​

Max record size

Kinesis has a maximum record size of 1 MB. If Kinesis rejects a message due to size limitations, Sequin will retry it indefinitely.

This means that the failing message will be saved until you take appropriate action and/or manually discard the message in the Sequin console.

For example, if one particular column caused the issue, you could define a transform function which removed that column from the message payload.

Then, after attaching that transform to your Kinesis sink, the next retry of the oversize message (automatic or manual) should succeed.

​

Ordering

When you configure a Kinesis sink, you can specify message grouping behavior. The grouping value is used as the partition key for records put into Kinesis. Records with the same key are ordered within a shard.

The default group for a message is the source row’s primary key(s). You can override this by specifying one or more columns to use for grouping.

Sequin will order the delivery of messages with the same group according to their commit timestamp.

​

Debugging

You can view the status of your Kinesis sink in the Sequin web console.

On the “Messages” tab, you can see which messages are in-flight to Kinesis, which messages Sequin is unable to deliver, and recently delivered messages.

Messages that Sequin is unable to deliver will have a “Deliver count” greater than 1. You can click on a message to see more details, including the last error response received from Kinesis.