sequin.yaml - Sequin

Overview

Configure Sequin resources like databases, sinks, and HTTP endpoints using YAML. You can provide YAML configuration to Sequin in three ways:

Via a configuration file using the CONFIG_FILE_PATH environment variable
Directly as base64-encoded YAML using the CONFIG_FILE_YAML environment variable
Via the Sequin CLI using the sequin config export|plan|apply command group

Schema

Account configuration

account:
  name: "account-name"  # Required for self-hosted deployments

Creating accounts is only supported on self-hosted Sequin.

User configuration

users:
  - email: "user@example.com"    # Required
    password: "userpassword123"  # Required

API token configuration

You can create API tokens for your account. This is intended for use in development or CI/CD workflows:

api_tokens:
  - name: "mytoken"
    token: "secret"

Creating API tokens is only supported on self-hosted Sequin.

Database configuration

databases:
  - name: "my-database"          # Required, unique identifier
    username: "postgres"         # Default: postgres
    password: "postgres"         # Default: postgres
    hostname: "localhost"        # Required
    port: 5432                  # Default: 5432
    database: "my_database"     # Required
    pool_size: 10               # Default: 3
    slot:
      name: "sequin_slot"        # Required
      create_if_not_exists: false   # Optional, default: false
    publication:
      name: "sequin_pub"         # Required
      create_if_not_exists: false   # Optional, default: false
      init_sql: |-               # Optional, SQL to run to create the publication. Defaults to create a publication for schema "public"
        create publication sequin_pub for tables in schema public with (publish_via_partition_root = true)
    await_database:             # Optional, configuration for database connection retry behavior
      timeout_ms: 30000         # Default: 30000 (30 seconds)
      interval_ms: 3000         # Default: 3000 (3 seconds)

These database entries tell Sequin which Postgres instances to capture changes from. They are separate from the Postgres database that Sequin itself uses, which is configured via the PG_* environment variables.

Replica database configuration

When connecting a replica to Sequin, Sequin also needs to connect to the primary database.

databases:
  - name: "my-database"          # Required, unique identifier
    username: "postgres"         # Default: postgres
    password: "postgres"         # Default: postgres
    hostname: "localhost"        # Required
    port: 5432                  # Default: 5432
    database: "my_database"     # Required
    pool_size: 10               # Default: 3
    slot:
      name: "sequin_slot"        # Required
    publication:
      name: "sequin_pub"         # Required
    await_database:             # Optional, configuration for database connection retry behavior
      timeout_ms: 30000         # Default: 30000 (30 seconds)
      interval_ms: 3000         # Default: 3000 (3 seconds)
    # Primary database connection details
    primary:
      username: "postgres"         # Default: postgres
      password: "postgres"         # Default: postgres
      hostname: "localhost"        # Required
      port: 5432                   # Default: 5432
      database: "my_database"      # Required

Replication slot and publication configuration

Sequin uses PostgreSQL’s logical replication to capture changes from your database. This requires a replication slot and a publication.

Replication slot

After following the instructions in the Sequin console to create a replication slot, you can indicate the name of the slot in the slot block. Alternatively, you can choose to have Sequin create the slot for you by setting create_if_not_exists to true:

slot:
  name: "sequin_slot"         # Required
  create_if_not_exists: true     # Optional, default: false

Replication slots are durable and can accumulate data if not in use (i.e. if Sequin is not running). So while Sequin can automatically create slots for you, slots will hang around until you drop them–even if you disconnect Sequin.

Publication

After following the instructions in the Sequin console to create a publication, you can indicate the name of the publication in the publication block. Alternatively, you can choose to have Sequin create the publication for you by setting create_if_not_exists to true:

publication:
  name: "sequin_pub"          # Required
  create_if_not_exists: true     # Optional, default: false
  init_sql: |-               # Optional, SQL to run to create the publication. Defaults to create a publication for schema "public"
    create publication sequin_pub for tables in schema public with (publish_via_partition_root = true)

When create_if_not_exists is true, Sequin will attempt to create the publication if it doesn’t exist. If you specify init_sql, Sequin will run that SQL to create the publication. Otherwise, Sequin will create a publication for the schema “public”.

Database connection retry

When creating a database, Sequin needs to be able to connect to the database in order to read the database’s schema. The await_database configuration allows you to control how Sequin attempts to connect to your database during startup. This option is most relevant in development environments, where you might be provisioning Sequin and your database at the same time. By default, Sequin will wait up to 30 seconds for the database to be ready and will retry every 3 seconds. You can customize this behavior by setting the timeout_ms and interval_ms options.

Sink configuration

A sink streams data from one or more tables to a destination (sink). All sinks share these configuration options:

sinks:
  - name: "my-sink"                # Required, unique name for this sink
    database: "my-database"        # Required, references database name
    source:
      include_schemas:             # Optional, schemas to include
        - "public"
      exclude_schemas:             # Optional, schemas to exclude
        - "private"
      include_tables:         # Optional, tables to include
        - "public.users"
      exclude_tables:         # Optional, tables to exclude
        - "public.background_jobs"
    status: "active"               # Optional: active or disabled, default: active
    enrichment: "my-enrichment"    # Optional, reference to an enrichment function name or "none"
    transform: "my-transform"      # Optional, reference to a transform function name or "none"
    filter: "my-filter"            # Optional, reference to a filter function name or "none"
    routing: "my-routing"          # Optional, reference to a routing function name or "none"
    annotations:                   # Optional, JSON object containing additional metadata that you've added
      my_data: "value"
    actions:                       # Optional, defaults to all
      - insert
      - update
      - delete
    message_grouping: true         # Optional, enable message grouping, default: true
    tables:                        # Optional, table specific configuration
      - name: "public.users"
        group_column_names:        # Optional, custom group column names
          - "account_id"
    batch_size: 1                  # Optional, messages per batch, default: 1
    destination:                   # Required, sink-specific configuration
      type: "..."                  # Required, sink type
      ...                          # Additional sink-specific options

Sink source

When no source configuration is specified, Sequin will capture changes from every table exposed by the publication. When new schemas or tables are added, they are automatically included. You can also specify schemas and tables to include or exclude from the stream.

Include only selected schemas or tables: Choose the schemas and tables you care about; changes from only those schemas and tables will be processed in your sink.
Exclude specific schemas or tables: Start with everything in the publication, then list the schemas or tables you want to omit. Future schemas and tables will be included in your sink unless they appear on your exclusion list.

For Sequin to stream changes from a table, your publication must include that table.

Sink actions

Additionally, you can specify actions to stream. By default, all actions (insert, update, delete) are included. read events are emitted during backfills and are always included.

Initial backfill

You can configure an initial backfill to run when a sink is created. This only applies when the sink is first created via YAML; updating an existing sink will not trigger a backfill. Simple syntax - full backfill of all tables in the source:

sinks:
  - name: "my-sink"
    database: "my-database"
    table: "public.users"
    initial_backfill: true

Advanced syntax - configure specific tables or partial backfills:

sinks:
  - name: "my-sink"
    database: "my-database"
    source:
      include_tables:
        - "public.users"
        - "public.orders"
    initial_backfill:
      - table: "public.users"          # Full backfill (default)
      - table: "public.orders"
        type: "partial"                 # Partial backfill
        sort_column: "updated_at"       # Column to sort by
        start_position: "2025-01-01T00:00:00Z"  # Start position

For partial backfills, you must specify both sort_column (the column to order by) and start_position (where to start the backfill from).

To trigger a backfill on an existing sink, use the Management API or the web console.

Message grouping

The message_grouping field controls whether messages are grouped for ordering purposes. When enabled (default), messages are grouped by primary key or custom grouping columns, ensuring ordered delivery within each group. When disabled, messages are not grouped, which may allow for higher throughput for some sinks.

true (default): Enable message grouping. Messages are grouped by primary key unless overridden by table-level group_column_names
false: Disable message grouping. Cannot be used with table-level group_column_names

sinks:
  - name: "ordered-sink"
    message_grouping: true  # Default, enables ordering
    tables:
      - name: "public.users"
        group_column_names: ["account_id"]  # Custom grouping

  - name: "high-throughput-sink"
    message_grouping: false
    tables:
      - name: "public.events"
        # group_column_names not allowed when message_grouping is false

Sink functions

The enrichment, transform, filter, and routing fields allow you to attach functions to your sink. Each field can be set to either:

The name of a function defined in the functions section
"none" to explicitly disable that type of function

For example, to attach a transform function to your sink:

functions:
  - name: "my-transform"
    type: "transform"
    code: |-
      def transform(action, record, changes, metadata) do
        %{
          id: record["id"],
          action: action
        }
      end

sinks:
  - name: "my-sink"
    transform: "my-transform"  # Reference the transform function
    ...

See Function configuration for more details on how to define functions.

Sink destinations

The destination configuration varies by sink type. Below are the configurations for each sink type:

Webhook sink

For sending changes to HTTP endpoints:

destination:
  type: "webhook"                 # Required
  http_endpoint: "endpoint-name"  # Required, references HTTP endpoint
  http_endpoint_path: "/custom"   # Optional, path to append to endpoint URL
  batch: false

Typesense sink

For indexing documents into a Typesense collection:

destination:
  type: "typesense"                # Required
  endpoint_url: "https://your-typesense-server:8108"  # Required
  collection_name: "products"      # Required
  api_key: "your-api-key"          # Required
  batch_size: 40                   # Optional, messages per batch, default: 40, max: 10000
  timeout_seconds: 5               # Optional, seconds to wait for response, default: 5, max: 300

Meilisearch sink

For indexing documents into a Meilisearch index:

destination:
  type: "meilisearch"              # Required
  endpoint_url: "http://your-meilisearch-server:7700"  # Required
  index_name: "products"           # Required
  primary_key: "id"                # Optional, primary key for the index, default: "id"
  api_key: "your-api-key"          # Required
  batch_size: 40                   # Optional, messages per batch, default: 40, max: 10000
  timeout_seconds: 5               # Optional, seconds to wait for response, default: 5, max: 300

Elasticsearch sink

For indexing documents into an Elasticsearch index:

destination:
  type: "elasticsearch"            # Required
  endpoint_url: "https://your-elasticsearch-cluster:9200"  # Required
  index_name: "products"           # Required
  auth_type: "api_key"             # Optional: api_key, basic, bearer, default: api_key
  auth_value: "your-auth-key"      # Required: API key, basic auth (username:password), or bearer token
  batch_size: 100                  # Optional, messages per batch, default: 100, max: 10000

Sequin Stream sink

Sequin Stream is a durable, scalable, and fault-tolerant message stream that you can use with Sequin in place of additional infrastructure like Kafka or SQS. For pulling changes via the Sequin Stream API:

destination:
  type: "sequin_stream"          # Required

Kafka sink

For publishing changes to Kafka topics:

destination:
  type: "kafka"                  # Required
  hosts: "localhost:9092"        # Required, comma-separated list of brokers
  topic: "my-topic"             # Required
  tls: false                    # Optional, enable TLS, default: false
  username: "kafka-user"        # Optional, for SASL authentication
  password: "kafka-pass"        # Optional, for SASL authentication
  sasl_mechanism: "plain"       # Optional: plain, scram_sha_256, scram_sha_512
  compression: "lz4"            # Optional: none (default), gzip, snappy, lz4, zstd
  batch_size: 50                # Optional, messages per batch

SQS sink

For sending changes to Amazon SQS queues: Using access keys:

destination:
  type: "sqs"                   # Required
  queue_url: "https://sqs.us-west-2.amazonaws.com/123/MyQueue.fifo" # Required
  region: "us-west-2"          # Optional, inferred from queue_url if omitted
  access_key_id: "AKIAXXXX"    # Required
  secret_access_key: "secret"   # Required

Using task role (recommended for AWS environments):

destination:
  type: "sqs"                   # Required
  queue_url: "https://sqs.us-west-2.amazonaws.com/123/MyQueue.fifo" # Required
  use_task_role: true           # Use ECS task role, EC2 instance profile, or EKS service account

Using LocalStack (for local development):

destination:
  type: "sqs"                   # Required
  queue_url: "http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000/my-queue"
  region: "us-east-1"
  use_emulator: true            # Enable emulator mode
  emulator_base_url: "http://localstack:4566"  # LocalStack endpoint
  access_key_id: "test"
  secret_access_key: "test"

For publishing changes to AWS SNS topics: Using access keys:

destination:
  type: "sns"                   # Required
  topic_arn: "arn:aws:sns:us-west-2:123456789012:my-topic" # Required, full SNS Topic ARN
  region: "us-west-2"          # Optional, inferred from topic_arn if omitted
  access_key_id: "AKIAXXXX"    # Required
  secret_access_key: "secret"   # Required
  is_fifo: false               # Optional, automatically detected from topic_arn

Using task role (recommended for AWS environments):

destination:
  type: "sns"                   # Required
  topic_arn: "arn:aws:sns:us-west-2:123456789012:my-topic" # Required, full SNS Topic ARN
  use_task_role: true           # Use ECS task role, EC2 instance profile, or EKS service account

Using LocalStack (for local development):

destination:
  type: "sns"                   # Required
  topic_arn: "arn:aws:sns:us-east-1:000000000000:my-topic"
  region: "us-east-1"
  use_emulator: true            # Enable emulator mode
  emulator_base_url: "http://localstack:4566"  # LocalStack endpoint
  access_key_id: "test"
  secret_access_key: "test"

Redis sink

For publishing changes to Redis streams:

destination:
  type: "redis_stream"          # Required
  host: "localhost"             # Required
  port: 6379                    # Required
  stream_key: "my-stream"       # Required
  database: 0                   # Optional, Redis database number, default: 0
  tls: false                    # Optional, enable TLS, default: false
  username: "redis-user"        # Optional, for authentication
  password: "redis-pass"        # Optional, for authentication

RabbitMQ sink

For publishing changes to RabbitMQ exchanges:

destination:
  type: "rabbitmq"              # Required
  host: "localhost"             # Required
  port: 5672                    # Required
  exchange: "my-exchange"       # Required
  username: "guest"             # Optional, default: guest
  password: "guest"             # Optional, default: guest
  virtual_host: "/"             # Optional, default: /
  tls: false                    # Optional, enable TLS, default: false

GCP PubSub sink

For publishing changes to Google Cloud Pub/Sub topics:

destination:
  type: "gcp_pubsub"           # Required
  project_id: "my-project"     # Required
  topic_id: "my-topic"         # Required
  credentials:                 # Required, GCP service account credentials
    type: "service_account"
    project_id: "my-project"
    private_key_id: "key123"
    private_key: "-----BEGIN PRIVATE KEY-----\nMIIE...\n-----END PRIVATE KEY-----\n"
    client_email: "my-service-account@my-project.iam.gserviceaccount.com"
    client_id: "123456789"
    auth_uri: "https://accounts.google.com/o/oauth2/auth"
    token_uri: "https://oauth2.googleapis.com/token"
    auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs"
    client_x509_cert_url: "https://www.googleapis.com/robot/v1/metadata/x509/my-service-account%40my-project.iam.gserviceaccount.com"
  use_emulator: false         # Optional, for local development
  emulator_base_url: "http://localhost:8085"  # Optional, required if use_emulator is true

The GCP PubSub sink requires a service account with permissions to publish to the specified topic. The credentials field should contain the JSON key file contents for a service account with the roles/pubsub.publisher role. Project ID must be between 6 and 30 characters, start with a letter, and contain only lowercase letters, numbers, and hyphens. Topic ID must be between 3 and 255 characters and match the pattern: [a-zA-Z][a-zA-Z0-9-_.~+%]*.

Azure Event Hub sink

For publishing changes to Azure Event Hubs:

destination:
  type: "azure_event_hub"          # Required
  namespace: "my-namespace"        # Required
  event_hub_name: "records"       # Required
  shared_access_key_name: "RootManageSharedAccessKey"  # Required
  shared_access_key: "your-shared-access-key"          # Required

NATS sink

For publishing changes to NATS:

destination:
  type: "nats"                    # Required
  host: "localhost"               # Required
  port: 4222                      # Required
  username: "nats_user"           # Optional
  password: "nats_password"       # Optional
  jwt: "your-jwt-token"          # Optional
  nkey_seed: "your-nkey-seed"    # Optional
  tls: false                     # Optional, enable TLS, default: false

S2 sink

For publishing changes to S2:

destination:
  type: "s2"                     # Required
  basin: "my-basin"              # Required
  stream: "my-stream"            # Required
  access_token: "my-access-token" # Required

Redis String sink

For storing changes as Redis strings:

destination:
  type: "redis_string"           # Required
  host: "localhost"              # Required
  port: 6379                     # Required
  database: 0                    # Optional, Redis database number, default: 0
  tls: false                     # Optional, enable TLS, default: false
  username: "redis_user"         # Optional, for authentication
  password: "redis_password"     # Optional, for authentication
  expire_ms: 3600000            # Optional, key expiration time in milliseconds
  mode: "static"                # Optional, key generation mode

Kinesis sink

For publishing changes to AWS Kinesis streams: Using access keys:

destination:
  type: "kinesis"                # Required
  stream_arn: "arn:aws:kinesis:us-east-1:123456789012:stream/my-stream"  # Required
  access_key_id: "AKIAXXXX"     # Required
  secret_access_key: "secret"    # Required

Using task role (recommended for AWS environments):

destination:
  type: "kinesis"                # Required
  stream_arn: "arn:aws:kinesis:us-east-1:123456789012:stream/my-stream"  # Required
  use_task_role: true            # Use ECS task role, EC2 instance profile, or EKS service account

Function configuration

Functions allow you to filter, transform, and route your data as it flows through Sequin. You can define functions at the top level of your configuration and then attach them to your sinks.

Function types

Sequin supports five types of functions:

Enrichment functions - Enrich your messages with additional data
Path functions - Extract data from a specific path in your message
Transform functions - Modify the structure of your messages
Filter functions - Filter which messages to process
Routing functions - Dynamically direct messages to different destinations

You can define functions in your configuration like this:

functions:
  # Enrichment function example
  - name: "my-enrichment-function"
    description: "Enrich with customer data"
    type: "enrichment"
    code: |-                          # Required for Enrichment functions
      SELECT
        u.id,
        a.name as account_name
      FROM
        users u
      JOIN
        accounts a on u.account_id = a.id
      WHERE
        u.id = ANY($1)

  # Path function example
  - name: "my-path-function"
    description: "Extract record"       # Optional
    type: "path"
    path: "record"                     # Required for path functions

  # Transform function example
  - name: "my-transform-function"
    description: "Extract ID and action"
    type: "transform"
    code: |-                          # Required for transform functions
      def transform(action, record, changes, metadata) do
        %{
          id: record["id"],
          action: action
        }
      end

  # Filter function example
  - name: "my-filter-function"
    description: "Filter VIP customers"
    type: "filter"
    code: |-                          # Required for filter functions
      def filter(action, record, changes, metadata) do
        record["customer_type"] == "VIP"
      end

  # Routing function example
  - name: "my-routing-function"
    description: "Route to REST API"
    type: "routing"
    sink_type: "webhook"              # Required, sink type to route to
    code: |-                          # Required for routing functions
      def route(action, record, changes, metadata) do
        %{
          method: "POST",
          endpoint_path: "/api/users/#{record["id"]}"
        }
      end

Functions via files

You can also define functions in separate files and reference them in your configuration.

functions:
  - name: "my-transform-function"
    type: "transform"
    file: "my-transform-function.exs" # Relative or absolute path to the file

When using this approach, the path is either absolute or relative to the location of the .yaml file.

Using functions in sinks

You can attach functions to your sinks using the transform, filter, and routing fields:

sinks:
  - name: "my-sink"
    enrichment: "my-enrichment"         # Reference an enrichment function
    transform: "my-transform-function"  # Reference a transform function
    filter: "my-filter-function"        # Reference a filter function
    routing: "my-routing-function"      # Reference a routing function
    ...

Each field can be set to either:

The name of a function defined in the functions section
"none" to explicitly disable that type of function

HTTP endpoint configuration

You can configure HTTP endpoints in three ways:

1. External URL

http_endpoints:
  - name: "external-endpoint"         # Required
    url: "https://api.example.com/webhook"  # Required
    headers:                          # Optional
      - key: "Content-Type"
        value: "application/json"
    encrypted_headers:                # Optional, for sensitive headers
      - key: "Authorization"
        value: "Bearer token123"

2. Local development endpoint

For local development, you can configure Sequin to connect to endpoints running on your local machine.

http_endpoints:
  - name: "local-endpoint"     # Required
    local: "true"             # Required
    path: "/webhook"          # Optional
    headers:                  # Optional
      - key: "X-Test"
        value: "test"
    encrypted_headers:        # Optional
      - key: "X-Secret"
        value: "secret123"

3. Webhook.site testing endpoint

http_endpoints:
  - name: "webhook-site"     # Required
    webhook.site: "true"     # Required

Change retention configuration

change_retentions:
  - name: "my-retention"                  # Required
    source_database: "source-db"          # Required
    source_table_schema: "public"         # Required
    source_table_name: "users"            # Required
    destination_database: "dest-db"       # Required
    destination_table_schema: "public"    # Required
    destination_table_name: "user_events" # Required
    actions:                             # Optional, defaults to all
      - insert
      - update
      - delete
    filters:                             # Optional
      - column_name: "status"
        operator: "="
        comparison_value: "active"
      - column_name: "metadata"          # JSONB column example
        field_path: "type.name"
        operator: "="
        comparison_value: "premium"
        field_type: "string"

Environment variable substitution

Sequin supports environment variable substitution in your YAML configuration files using the following syntax:

${VARIABLE_NAME:-default_value}

This allows you to:

Reference environment variables in your configuration
Provide default values when the environment variable is not set

For example:

databases:
  - name: "production-db"
    # Note quotes around the reference in case of special characters (see below)
    hostname: "${DB_HOST:-localhost}"
    # You can alternatively use a YAML heredoc, if your env variable might contain quotes (see below)
    password: |-
      ${DB_PASSWORD:-postgres}
    database: "${DB_NAME:-app_production}"

Best practices for environment variables in YAML

YAML has special characters that can affect parsing if they appear unquoted in your configuration. When using environment variables that might contain special characters (like :, -, {, }, [, ], etc.), it’s recommended to quote the entire reference:

# Safe pattern for values that might contain special characters
api_key: "${API_KEY}"
# Alternatively, use a heredoc if your string contains quotes
password: |-
  ${DB_PASSWORD}

This ensures that the substituted value is treated as a string literal regardless of its content, preventing YAML parsing errors.

Using with dotenv

You can use tools like dotenv to load environment variables from a .env file before running Sequin commands:

# Load environment variables from .env file and apply configuration
dotenv -- sequin config apply

This is particularly useful for:

Managing environment-specific configurations
Keeping sensitive values out of your YAML files
Simplifying deployment across different environments

Previewing interpolated YAML

You can preview how your environment variables will be interpolated into your configuration before applying it:

# View the interpolated YAML with environment variables substituted
sequin config interpolate my-config.yaml

# Write the interpolated YAML to a file
sequin config interpolate my-config.yaml --output interpolated.yaml

This helps verify that your configuration will be processed correctly, especially when using default values or complex environment variable patterns.

Example configuration

Here’s a complete example combining multiple resources:

account:
  name: "my-account"

users:
  - email: "admin@example.com"
    password: "adminpass123"

databases:
  - name: "production-db"
    hostname: "prod.db.example.com"
    database: "app_production"
    slot:
      name: "sequin_slot"
      create_if_not_exists: true
    publication:
      name: "sequin_pub"
      create_if_not_exists: true
      schemas: ["public"]

http_endpoints:
  - name: "webhook-endpoint"
    url: "https://api.example.com/webhook"
    encrypted_headers:
      - key: "Authorization"
        value: "Bearer token123"

sinks:
  - name: "user-changes"
    database: "production-db"
    schema: "public"
    destination:
      type: "webhook"
      http_endpoint: "webhook-endpoint"

change_retentions:
  - name: "user-audit"
    source_database: "production-db"
    source_table_schema: "public"
    source_table_name: "users"
    destination_database: "production-db"
    destination_table_schema: "audit"
    destination_table_name: "user_events"

YAML Anchors

YAML anchors allow you to reuse YAML configuration across multiple resources. A common pattern is to send the same data stream to multiple destinations for different purposes - for example, audit logging and microservice communication.

databases:
  - name: "production-db"
    hostname: "prod.db.example.com"
    database: "app_production"
    slot:
      name: "sequin_slot"
    publication:
      name: "sequin_pub"

# Create a reusable base sink configuration
base_sink_config: &base_sink_config
  database: "production-db"
  schema: "public"  # Stream all tables in the public schema
  transform: "sanitize-pii" # Sanitize PII data for all sinks

sinks:
  # Kafka sink for audit logging and compliance
  - <<: *base_sink_config
    name: "audit-logging"
    destination:
      type: "kafka"
      hosts: "kafka-cluster.company.com:9092"
      topic: "audit-events"
      tls: true
      username: "audit-service"
      password: "audit-password"
      sasl_mechanism: "scram_sha_256"

  # SQS sink for microservice communication
  - <<: *base_sink_config
    name: "microservice-events"
    destination:
      type: "sqs"
      queue_url: "https://sqs.us-west-2.amazonaws.com/123456789012/service-events.fifo"
      access_key_id: "${AWS_ACCESS_KEY_ID}"
      secret_access_key: "${AWS_SECRET_ACCESS_KEY}"

This setup streams changes from all tables in your public schema to both Kafka (for audit logging) and SQS (for microservice communication), using YAML anchors to avoid duplicating the common sink configuration.

​Overview

​Schema

​Account configuration

​User configuration

​API token configuration

​Database configuration

​Replica database configuration

​Replication slot and publication configuration

Replication slot

Publication

​Database connection retry

​Sink configuration

​Sink source

​Sink actions

​Initial backfill

​Message grouping

​Sink functions

​Sink destinations

​Webhook sink

​Typesense sink

​Meilisearch sink

​Elasticsearch sink

​Sequin Stream sink

​Kafka sink

​SQS sink

​SNS sink

​Redis sink

​RabbitMQ sink

​GCP PubSub sink

​Azure Event Hub sink

​NATS sink

​S2 sink

​Redis String sink

​Kinesis sink

​Function configuration

​Function types

​Functions via files

​Using functions in sinks

​HTTP endpoint configuration

​1. External URL

​2. Local development endpoint

​3. Webhook.site testing endpoint

​Change retention configuration

​Environment variable substitution

​Best practices for environment variables in YAML

​Using with dotenv

​Previewing interpolated YAML

​Example configuration

​YAML Anchors

Overview

Schema

Account configuration

User configuration

API token configuration

Database configuration

Replica database configuration

Replication slot and publication configuration

Database connection retry

Sink configuration

Sink source

Sink actions

Initial backfill

Message grouping

Sink functions

Sink destinations

Webhook sink

Typesense sink

Meilisearch sink

Elasticsearch sink

Sequin Stream sink

Kafka sink

SQS sink

SNS sink

Redis sink

RabbitMQ sink

GCP PubSub sink

Azure Event Hub sink

NATS sink

S2 sink

Redis String sink

Kinesis sink

Function configuration

Function types

Functions via files

Using functions in sinks

HTTP endpoint configuration

1. External URL

2. Local development endpoint

3. Webhook.site testing endpoint

Change retention configuration

Environment variable substitution

Best practices for environment variables in YAML

Using with dotenv

Previewing interpolated YAML

Example configuration

YAML Anchors