​
Overview

Configure Sequin resources like databases, sinks, and HTTP endpoints using YAML.

You can provide YAML configuration to Sequin in three ways:

  1. Via a configuration file using the CONFIG_FILE_PATH environment variable
  2. Directly as base64-encoded YAML using the CONFIG_FILE_YAML environment variable
  3. Via the Sequin CLI using the sequin config export|plan|apply command group

​
Schema

​
Account configuration

account:
  name: "account-name"  # Required for self-hosted deployments

Creating accounts is only supported on self-hosted Sequin.

​
User configuration

users:
  - email: "user@example.com"    # Required
    password: "userpassword123"  # Required

​
API token configuration

You can create API tokens for your account. This is intended for use in development or CI/CD workflows:

api_tokens:
  - name: "mytoken"
    token: "secret"

Creating API tokens is only supported on self-hosted Sequin.

​
Database configuration

databases:
  - name: "my-database"          # Required, unique identifier
    username: "postgres"         # Default: postgres
    password: "postgres"         # Default: postgres
    hostname: "localhost"        # Required
    port: 5432                  # Default: 5432
    database: "my_database"     # Required
    pool_size: 10               # Default: 3
    slot:
      name: "sequin_slot"        # Required
      create_if_not_exists: false   # Optional, default: false
    publication:
      name: "sequin_pub"         # Required
      create_if_not_exists: false   # Optional, default: false
      init_sql: |-               # Optional, SQL to run to create the publication. Defaults to create a publication for schema "public"
        create publication sequin_pub for tables in schema public with (publish_via_partition_root = true)
    await_database:             # Optional, configuration for database connection retry behavior
      timeout_ms: 30000         # Default: 30000 (30 seconds)
      interval_ms: 3000         # Default: 3000 (3 seconds)

These database entries tell Sequin which Postgres instances to capture changes from. They are separate from the Postgres database that Sequin itself uses, which is configured via the PG_* environment variables.

​
Replica database configuration

When connecting a replica to Sequin, Sequin also needs to connect to the primary database.

databases:
  - name: "my-database"          # Required, unique identifier
    username: "postgres"         # Default: postgres
    password: "postgres"         # Default: postgres
    hostname: "localhost"        # Required
    port: 5432                  # Default: 5432
    database: "my_database"     # Required
    pool_size: 10               # Default: 3
    slot:
      name: "sequin_slot"        # Required
    publication:
      name: "sequin_pub"         # Required
    await_database:             # Optional, configuration for database connection retry behavior
      timeout_ms: 30000         # Default: 30000 (30 seconds)
      interval_ms: 3000         # Default: 3000 (3 seconds)
    # Primary database connection details
    primary:
      username: "postgres"         # Default: postgres
      password: "postgres"         # Default: postgres
      hostname: "localhost"        # Required
      port: 5432                   # Default: 5432
      database: "my_database"      # Required

​
Replication slot and publication configuration

Sequin uses PostgreSQL’s logical replication to capture changes from your database. This requires a replication slot and a publication.

Replication slot

After following the instructions in the Sequin console to create a replication slot, you can indicate the name of the slot in the slot block. Alternatively, you can choose to have Sequin create the slot for you by setting create_if_not_exists to true:

slot:
  name: "sequin_slot"         # Required
  create_if_not_exists: true     # Optional, default: false

Replication slots are durable and can accumulate data if not in use (i.e. if Sequin is not running). So while Sequin can automatically create slots for you, slots will hang around until you drop them–even if you disconnect Sequin.

Publication

After following the instructions in the Sequin console to create a publication, you can indicate the name of the publication in the publication block. Alternatively, you can choose to have Sequin create the publication for you by setting create_if_not_exists to true:

publication:
  name: "sequin_pub"          # Required
  create_if_not_exists: true     # Optional, default: false
  init_sql: |-               # Optional, SQL to run to create the publication. Defaults to create a publication for schema "public"
    create publication sequin_pub for tables in schema public with (publish_via_partition_root = true)

When create_if_not_exists is true, Sequin will attempt to create the publication if it doesn’t exist. If you specify init_sql, Sequin will run that SQL to create the publication. Otherwise, Sequin will create a publication for the schema “public”.

​
Database connection retry

When creating a database, Sequin needs to be able to connect to the database in order to read the database’s schema. The await_database configuration allows you to control how Sequin attempts to connect to your database during startup. This option is most relevant in development environments, where you might be provisioning Sequin and your database at the same time.

By default, Sequin will wait up to 30 seconds for the database to be ready and will retry every 3 seconds. You can customize this behavior by setting the timeout_ms and interval_ms options.

​
Sink configuration

A sink streams data from one or more tables to a destination (sink). All sinks share these configuration options:

sinks:
  - name: "my-sink"                # Required, unique name for this sink
    database: "my-database"        # Required, references database name
    source:
      include_schemas:             # Optional, schemas to include
        - "public"
      exclude_schemas:             # Optional, schemas to exclude
        - "private"
      include_tables:         # Optional, tables to include
        - "public.users"
      exclude_tables:         # Optional, tables to exclude
        - "public.background_jobs"
    status: "active"               # Optional: active or disabled, default: active
    transform: "my-transform"      # Optional, reference to a transform function name or "none"
    filter: "my-filter"            # Optional, reference to a filter function name or "none"
    routing: "my-routing"          # Optional, reference to a routing function name or "none"
    annotations:                   # Optional, JSON object containing additional metadata that you've added
      my_data: "value"
    actions:                       # Optional, defaults to all
      - insert
      - update
      - delete
    message_grouping: true         # Optional, enable message grouping, default: true
    tables:                        # Optional, table specific configuration
      - name: "public.users"
        group_column_names:        # Optional, custom group column names
          - "account_id"
    batch_size: 1                  # Optional, messages per batch, default: 1
    destination:                   # Required, sink-specific configuration
      type: "..."                  # Required, sink type
      ...                          # Additional sink-specific options

​
Sink source

When no source configuration is specified, Sequin will capture changes from every table exposed by the publication. When new schemas or tables are added, they are automatically included.

You can also specify schemas and tables to include or exclude from the stream.

  • Include only selected schemas or tables: Choose the schemas and tables you care about; changes from only those schemas and tables will be processed in your sink.
  • Exclude specific schemas or tables: Start with everything in the publication, then list the schemas or tables you want to omit. Future schemas and tables will be included in your sink unless they appear on your exclusion list.

For Sequin to stream changes from a table, your publication must include that table.

​
Sink actions

Additionally, you can specify actions to stream. By default, all actions (insert, update, delete) are included. read events are emitted during backfills and are always included.

​
Message grouping

The message_grouping field controls whether messages are grouped for ordering purposes. When enabled (default), messages are grouped by primary key or custom grouping columns, ensuring ordered delivery within each group. When disabled, messages are not grouped, which may allow for higher throughput for some sinks.

  • true (default): Enable message grouping. Messages are grouped by primary key unless overridden by table-level group_column_names
  • false: Disable message grouping. Cannot be used with table-level group_column_names
sinks:
  - name: "ordered-sink"
    message_grouping: true  # Default, enables ordering
    tables:
      - name: "public.users"
        group_column_names: ["account_id"]  # Custom grouping

  - name: "high-throughput-sink"
    message_grouping: false
    tables:
      - name: "public.events"
        # group_column_names not allowed when message_grouping is false

​
Sink functions

The transform, filter, and routing fields allow you to attach functions to your sink. Each field can be set to either:

  • The name of a function defined in the functions section
  • "none" to explicitly disable that type of function

For example, to attach a transform function to your sink:

functions:
  - name: "my-transform"
    type: "transform"
    code: |-
      def transform(action, record, changes, metadata) do
        %{
          id: record["id"],
          action: action
        }
      end

sinks:
  - name: "my-sink"
    transform: "my-transform"  # Reference the transform function
    ...

See Function configuration for more details on how to define functions.

​
Sink destinations

The destination configuration varies by sink type. Below are the configurations for each sink type:

​
Webhook sink

For sending changes to HTTP endpoints:

destination:
  type: "webhook"                 # Required
  http_endpoint: "endpoint-name"  # Required, references HTTP endpoint
  http_endpoint_path: "/custom"   # Optional, path to append to endpoint URL
  batch: false

​
Typesense sink

For indexing documents into a Typesense collection:

destination:
  type: "typesense"                # Required
  endpoint_url: "https://your-typesense-server:8108"  # Required
  collection_name: "products"      # Required
  api_key: "your-api-key"          # Required
  batch_size: 40                   # Optional, messages per batch, default: 40, max: 10000
  timeout_seconds: 5               # Optional, seconds to wait for response, default: 5, max: 300

​
Meilisearch sink

For indexing documents into a Meilisearch index:

destination:
  type: "meilisearch"              # Required
  endpoint_url: "http://your-meilisearch-server:7700"  # Required
  index_name: "products"           # Required
  primary_key: "id"                # Optional, primary key for the index, default: "id"
  api_key: "your-api-key"          # Required
  batch_size: 40                   # Optional, messages per batch, default: 40, max: 10000
  timeout_seconds: 5               # Optional, seconds to wait for response, default: 5, max: 300

​
Elasticsearch sink

For indexing documents into an Elasticsearch index:

destination:
  type: "elasticsearch"            # Required
  endpoint_url: "https://your-elasticsearch-cluster:9200"  # Required
  index_name: "products"           # Required
  auth_type: "api_key"             # Optional: api_key, basic, bearer, default: api_key
  auth_value: "your-auth-key"      # Required: API key, basic auth (username:password), or bearer token
  batch_size: 100                  # Optional, messages per batch, default: 100, max: 10000

​
Sequin Stream sink

Sequin Stream is a durable, scalable, and fault-tolerant message stream that you can use with Sequin in place of additional infrastructure like Kafka or SQS.

For pulling changes via the Sequin Stream API:

destination:
  type: "sequin_stream"          # Required

​
Kafka sink

For publishing changes to Kafka topics:

destination:
  type: "kafka"                  # Required
  hosts: "localhost:9092"        # Required, comma-separated list of brokers
  topic: "my-topic"             # Required
  tls: false                    # Optional, enable TLS, default: false
  username: "kafka-user"        # Optional, for SASL authentication
  password: "kafka-pass"        # Optional, for SASL authentication
  sasl_mechanism: "plain"       # Optional: plain, scram_sha_256, scram_sha_512
  batch_size: 50                # Optional, messages per batch

​
SQS sink

For sending changes to Amazon SQS queues:

destination:
  type: "sqs"                   # Required
  queue_url: "https://sqs.us-west-2.amazonaws.com/123/MyQueue.fifo" # Required
  region: "us-west-2"          # Optional, inferred from queue_url if omitted
  access_key_id: "AKIAXXXX"    # Required
  secret_access_key: "secret"   # Required

​
SNS sink

For publishing changes to AWS SNS topics:

destination:
  type: "sns"                   # Required
  topic_arn: "arn:aws:sns:us-west-2:123456789012:my-topic" # Required, full SNS Topic ARN
  region: "us-west-2"          # Optional, inferred from topic_arn if omitted
  access_key_id: "AKIAXXXX"    # Required
  secret_access_key: "secret"   # Required
  is_fifo: false               # Optional, automatically detected from topic_arn

​
Redis sink

For publishing changes to Redis streams:

destination:
  type: "redis_stream"          # Required
  host: "localhost"             # Required
  port: 6379                    # Required
  stream_key: "my-stream"       # Required
  database: 0                   # Optional, Redis database number, default: 0
  tls: false                    # Optional, enable TLS, default: false
  username: "redis-user"        # Optional, for authentication
  password: "redis-pass"        # Optional, for authentication

​
RabbitMQ sink

For publishing changes to RabbitMQ exchanges:

destination:
  type: "rabbitmq"              # Required
  host: "localhost"             # Required
  port: 5672                    # Required
  exchange: "my-exchange"       # Required
  username: "guest"             # Optional, default: guest
  password: "guest"             # Optional, default: guest
  virtual_host: "/"             # Optional, default: /
  tls: false                    # Optional, enable TLS, default: false

​
GCP PubSub sink

For publishing changes to Google Cloud Pub/Sub topics:

destination:
  type: "gcp_pubsub"           # Required
  project_id: "my-project"     # Required
  topic_id: "my-topic"         # Required
  credentials:                 # Required, GCP service account credentials
    type: "service_account"
    project_id: "my-project"
    private_key_id: "key123"
    private_key: "-----BEGIN PRIVATE KEY-----\nMIIE...\n-----END PRIVATE KEY-----\n"
    client_email: "my-service-account@my-project.iam.gserviceaccount.com"
    client_id: "123456789"
    auth_uri: "https://accounts.google.com/o/oauth2/auth"
    token_uri: "https://oauth2.googleapis.com/token"
    auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs"
    client_x509_cert_url: "https://www.googleapis.com/robot/v1/metadata/x509/my-service-account%40my-project.iam.gserviceaccount.com"
  use_emulator: false         # Optional, for local development
  emulator_base_url: "http://localhost:8085"  # Optional, required if use_emulator is true

The GCP PubSub sink requires a service account with permissions to publish to the specified topic. The credentials field should contain the JSON key file contents for a service account with the roles/pubsub.publisher role.

Project ID must be between 6 and 30 characters, start with a letter, and contain only lowercase letters, numbers, and hyphens. Topic ID must be between 3 and 255 characters and match the pattern: [a-zA-Z][a-zA-Z0-9-_.~+%]*.

​
Azure Event Hub sink

For publishing changes to Azure Event Hubs:

destination:
  type: "azure_event_hub"          # Required
  namespace: "my-namespace"        # Required
  event_hub_name: "records"       # Required
  shared_access_key_name: "RootManageSharedAccessKey"  # Required
  shared_access_key: "your-shared-access-key"          # Required

​
NATS sink

For publishing changes to NATS:

destination:
  type: "nats"                    # Required
  host: "localhost"               # Required
  port: 4222                      # Required
  username: "nats_user"           # Optional
  password: "nats_password"       # Optional
  jwt: "your-jwt-token"          # Optional
  nkey_seed: "your-nkey-seed"    # Optional
  tls: false                     # Optional, enable TLS, default: false

​
S2 sink

For publishing changes to S2:

destination:
  type: "s2"                     # Required
  basin: "my-basin"              # Required
  stream: "my-stream"            # Required
  access_token: "my-access-token" # Required

​
Redis String sink

For storing changes as Redis strings:

destination:
  type: "redis_string"           # Required
  host: "localhost"              # Required
  port: 6379                     # Required
  database: 0                    # Optional, Redis database number, default: 0
  tls: false                     # Optional, enable TLS, default: false
  username: "redis_user"         # Optional, for authentication
  password: "redis_password"     # Optional, for authentication
  expire_ms: 3600000            # Optional, key expiration time in milliseconds
  mode: "static"                # Optional, key generation mode

​
Kinesis sink

For publishing changes to AWS Kinesis streams:

destination:
  type: "kinesis"                # Required
  stream_arn: "arn:aws:kinesis:us-east-1:123456789012:stream/my-stream"  # Required
  access_key_id: "AKIAXXXX"     # Required
  secret_access_key: "secret"    # Required

​
Function configuration

Functions allow you to filter, transform, and route your data as it flows through Sequin. You can define functions at the top level of your configuration and then attach them to your sinks.

​
Function types

Sequin supports four types of functions:

  1. Path functions - Extract data from a specific path in your message
  2. Transform functions - Modify the structure of your messages
  3. Filter functions - Filter which messages to process
  4. Routing functions - Dynamically direct messages to different destinations

You can define functions in your configuration like this:

functions:
  # Path function example
  - name: "my-path-function"
    description: "Extract record"       # Optional
    type: "path"
    path: "record"                     # Required for path functions

  # Transform function example
  - name: "my-transform-function"
    description: "Extract ID and action"
    type: "transform"
    code: |-                          # Required for transform functions
      def transform(action, record, changes, metadata) do
        %{
          id: record["id"],
          action: action
        }
      end

  # Filter function example
  - name: "my-filter-function"
    description: "Filter VIP customers"
    type: "filter"
    code: |-                          # Required for filter functions
      def filter(action, record, changes, metadata) do
        record["customer_type"] == "VIP"
      end

  # Routing function example
  - name: "my-routing-function"
    description: "Route to REST API"
    type: "routing"
    sink_type: "webhook"              # Required, sink type to route to
    code: |-                          # Required for routing functions
      def route(action, record, changes, metadata) do
        %{
          method: "POST",
          endpoint_path: "/api/users/#{record["id"]}"
        }
      end

​
Using functions in sinks

You can attach functions to your sinks using the transform, filter, and routing fields:

sinks:
  - name: "my-sink"
    transform: "my-transform-function"  # Reference a transform function
    filter: "my-filter-function"       # Reference a filter function
    routing: "my-routing-function"     # Reference a routing function
    ...

Each field can be set to either:

  • The name of a function defined in the functions section
  • "none" to explicitly disable that type of function

​
HTTP endpoint configuration

You can configure HTTP endpoints in three ways:

​
1. External URL

http_endpoints:
  - name: "external-endpoint"         # Required
    url: "https://api.example.com/webhook"  # Required
    headers:                          # Optional
      - key: "Content-Type"
        value: "application/json"
    encrypted_headers:                # Optional, for sensitive headers
      - key: "Authorization"
        value: "Bearer token123"

​
2. Local development endpoint

Sequin Cloud offers tunneling to your local machine for development. Learn more about tunneling in the CLI documentation.

http_endpoints:
  - name: "local-endpoint"     # Required
    local: "true"             # Required
    path: "/webhook"          # Optional
    headers:                  # Optional
      - key: "X-Test"
        value: "test"
    encrypted_headers:        # Optional
      - key: "X-Secret"
        value: "secret123"

​
3. Webhook.site testing endpoint

http_endpoints:
  - name: "webhook-site"     # Required
    webhook.site: "true"     # Required

​
Change retention configuration

change_retentions:
  - name: "my-retention"                  # Required
    source_database: "source-db"          # Required
    source_table_schema: "public"         # Required
    source_table_name: "users"            # Required
    destination_database: "dest-db"       # Required
    destination_table_schema: "public"    # Required
    destination_table_name: "user_events" # Required
    actions:                             # Optional, defaults to all
      - insert
      - update
      - delete
    filters:                             # Optional
      - column_name: "status"
        operator: "="
        comparison_value: "active"
      - column_name: "metadata"          # JSONB column example
        field_path: "type.name"
        operator: "="
        comparison_value: "premium"
        field_type: "string"

​
Environment variable substitution

Sequin supports environment variable substitution in your YAML configuration files using the following syntax:

${VARIABLE_NAME:-default_value}

This allows you to:

  • Reference environment variables in your configuration
  • Provide default values when the environment variable is not set

For example:

databases:
  - name: "production-db"
    # Note quotes around the reference in case of special characters (see below)
    hostname: "${DB_HOST:-localhost}"
    # You can alternatively use a YAML heredoc, if your env variable might contain quotes (see below)
    password: |-
      ${DB_PASSWORD:-postgres}
    database: "${DB_NAME:-app_production}"

​
Best practices for environment variables in YAML

YAML has special characters that can affect parsing if they appear unquoted in your configuration. When using environment variables that might contain special characters (like :, -, {, }, [, ], etc.), it’s recommended to quote the entire reference:

# Safe pattern for values that might contain special characters
api_key: "${API_KEY}"
# Alternatively, use a heredoc if your string contains quotes
password: |-
  ${DB_PASSWORD}

This ensures that the substituted value is treated as a string literal regardless of its content, preventing YAML parsing errors.

​
Using with dotenv

You can use tools like dotenv to load environment variables from a .env file before running Sequin commands:

# Load environment variables from .env file and apply configuration
dotenv -- sequin config apply

This is particularly useful for:

  • Managing environment-specific configurations
  • Keeping sensitive values out of your YAML files
  • Simplifying deployment across different environments

​
Previewing interpolated YAML

You can preview how your environment variables will be interpolated into your configuration before applying it:

# View the interpolated YAML with environment variables substituted
sequin config interpolate my-config.yaml

# Write the interpolated YAML to a file
sequin config interpolate my-config.yaml --output interpolated.yaml

This helps verify that your configuration will be processed correctly, especially when using default values or complex environment variable patterns.

​
Example configuration

Here’s a complete example combining multiple resources:

account:
  name: "my-account"

users:
  - email: "admin@example.com"
    password: "adminpass123"

databases:
  - name: "production-db"
    hostname: "prod.db.example.com"
    database: "app_production"
    slot:
      name: "sequin_slot"
      create_if_not_exists: true
    publication:
      name: "sequin_pub"
      create_if_not_exists: true
      schemas: ["public"]

http_endpoints:
  - name: "webhook-endpoint"
    url: "https://api.example.com/webhook"
    encrypted_headers:
      - key: "Authorization"
        value: "Bearer token123"

sinks:
  - name: "user-changes"
    database: "production-db"
    schema: "public"
    destination:
      type: "webhook"
      http_endpoint: "webhook-endpoint"

change_retentions:
  - name: "user-audit"
    source_database: "production-db"
    source_table_schema: "public"
    source_table_name: "users"
    destination_database: "production-db"
    destination_table_schema: "audit"
    destination_table_name: "user_events"

​
YAML Anchors

YAML anchors allow you to reuse YAML configuration across multiple resources. A common pattern is to send the same data stream to multiple destinations for different purposes - for example, audit logging and microservice communication.

databases:
  - name: "production-db"
    hostname: "prod.db.example.com"
    database: "app_production"
    slot:
      name: "sequin_slot"
    publication:
      name: "sequin_pub"

# Create a reusable base sink configuration
base_sink_config: &base_sink_config
  database: "production-db"
  schema: "public"  # Stream all tables in the public schema
  transform: "sanitize-pii" # Sanitize PII data for all sinks

sinks:
  # Kafka sink for audit logging and compliance
  - <<: *base_sink_config
    name: "audit-logging"
    destination:
      type: "kafka"
      hosts: "kafka-cluster.company.com:9092"
      topic: "audit-events"
      tls: true
      username: "audit-service"
      password: "audit-password"
      sasl_mechanism: "scram_sha_256"

  # SQS sink for microservice communication
  - <<: *base_sink_config
    name: "microservice-events"
    destination:
      type: "sqs"
      queue_url: "https://sqs.us-west-2.amazonaws.com/123456789012/service-events.fifo"
      access_key_id: "${AWS_ACCESS_KEY_ID}"
      secret_access_key: "${AWS_SECRET_ACCESS_KEY}"

This setup streams changes from all tables in your public schema to both Kafka (for audit logging) and SQS (for microservice communication), using YAML anchors to avoid duplicating the common sink configuration.