The SQS sink sends messages to an Amazon SQS queue.
This is the reference for the SQS sink. See the quickstart for a step-by-step walkthrough or the how-to guide for an explanation of how to use the SQS sink.

Configuration

  • Queue URL The URL of your Amazon SQS queue. Must be in the format: https://sqs.<region>.amazonaws.com/<account-id>/<queue-name>. For FIFO queues, the queue name must end with .fifo.

IAM permissions

Sequin requires the following AWS permissions to send messages to SQS:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:SendMessageBatch",
        "sqs:GetQueueAttributes",
        "sqs:GetQueueUrl",
        "sqs:ListQueues",
        "sqs:ListQueueTags",
        "sqs:ChangeMessageVisibility"
      ],
      "Resource": "<your-queue-arn>"
    }
  ]
}
Replace <your-queue-arn> with your actual queue ARN (e.g., arn:aws:sqs:us-east-1:123456789012:my-queue).

Configuration examples

Using access keys

sinks:
  - name: "sqs-sink"
    batch_size: 10
    destination:
      type: "sqs"
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue"
      access_key_id: "AKIA..."
      secret_access_key: "your-secret-key"
sinks:
  - name: "sqs-sink"
    batch_size: 10
    destination:
      type: "sqs"
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/my-queue"
      use_task_role: true
When using use_task_role: true, ensure your ECS task role, EC2 instance profile, or EKS service account has the required SQS permissions.

Message format

Sequin sends messages to SQS as JSON. You can find the shape of the messages in the messages reference.

Retry behavior

If Sequin is unable to deliver a message to SQS, it will retry the message indefinitely. Sequin will exponentially back off the retry interval, with a maximum backoff of roughly 3 minutes.

Max message size

SQS allows for a max message size of 256 KB. If SQS rejects a message, Sequin will cancel the message’s delivery.
If you want to see logging/alerting for this situation, please upvote the corresponding issue.

Grouping and ordering

You can control message grouping behavior using the message_grouping configuration option:
  • message_grouping: true (default): Enable message grouping for ordered delivery
  • message_grouping: false: Disable message grouping for maximum throughput
When message_grouping is enabled, Sequin will set the MessageGroupId on SQS messages if you’re using a FIFO queue. The default message group for a message is the source row’s primary key(s). You can override this by specifying one or more columns to use for message grouping at the table level. When message_grouping is disabled, no MessageGroupId is set, allowing for higher throughput but without ordering guarantees. Sequin will order the delivery of messages with the same group according to their commit timestamp.
sinks:
  - name: "sqs-ordered"
    message_grouping: true  # Enable grouping
    batch_size: 10
    tables:
      - name: "public.orders"
        group_column_names: ["customer_id"]  # Group by customer instead of PK
    destination:
      type: "sqs"
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/orders.fifo"
      use_task_role: true

  - name: "sqs-high-throughput"
    message_grouping: false  # Disable grouping for max throughput
    batch_size: 10
    destination:
      type: "sqs"
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/events"
      use_task_role: true

FIFO vs standard queues

FIFO queues are generally recommended. They ensure that messages pertaining to the same row are processed in order.

Routing

The AWS SQS sink supports dynamic routing of the queue_url with routing functions. Example routing function:
def route(action, record, changes, metadata) do
  %{
    queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/#{metadata.table_name}"
  }
end
When not using a routing function, messages will be published to the queue specified in the sink configuration.

Running SQS locally

You can run SQS locally using LocalStack.
docker run -d -p 4566:4566 localstack/localstack
Then, include the use_emulator: true and emulator_base_url: "http://localstack:4566" configuration to send messages to the local SQS queue:
sinks:
  - name: "demo-to-sqs"
    database: "e2e-database"
    table: "public.sqs_test_table"
    batch_size: 10
    status: "active"
    actions:
      - insert
      - update
      - delete
    destination:
      type: "sqs"
      queue_url: "http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/demo-queue"
      region: "us-east-1"
      use_emulator: true
      emulator_base_url: "http://localstack:4566"
      access_key_id: "test"
      secret_access_key: "test"

Debugging

You can view the status of your SQS sink in the Sequin web console. On the “Messages” tab, you can see which messages are in-flight to SQS, which messages Sequin is unable to deliver, and recently delivered messages. Messages that Sequin is unable to deliver will have a “Deliver count” greater than 1. You can click on a message to see more details, including the last error response received from SQS.