​
Overview

Configure Sequin resources like databases, sinks, and HTTP endpoints using YAML.

You can provide YAML configuration to Sequin in three ways:

  1. Via a configuration file using the CONFIG_FILE_PATH environment variable
  2. Directly as base64-encoded YAML using the CONFIG_FILE_YAML environment variable
  3. Via the Sequin CLI using the sequin config export|plan|apply command group

​
Schema

​
Account configuration

account:
  name: "account-name"  # Required for self-hosted deployments

Creating accounts is only supported on self-hosted Sequin.

​
User configuration

users:
  - email: "user@example.com"    # Required
    password: "userpassword123"  # Required

​
API token configuration

You can create API tokens for your account. This is intended for use in development or CI/CD workflows:

api_tokens:
  - name: "mytoken"
    token: "secret"

Creating API tokens is only supported on self-hosted Sequin.

​
Database configuration

databases:
  - name: "my-database"          # Required, unique identifier
    username: "postgres"         # Default: postgres
    password: "postgres"         # Default: postgres
    hostname: "localhost"        # Required
    port: 5432                  # Default: 5432
    database: "my_database"     # Required
    pool_size: 10               # Default: 3
    slot_name: "sequin_slot"    # Required
    publication_name: "sequin_pub" # Required
    await_database:             # Optional, configuration for database connection retry behavior
      timeout_ms: 30000         # Default: 30000 (30 seconds)
      interval_ms: 3000         # Default: 3000 (3 seconds)

​
Database connection retry

When creating a database, Sequin needs to be able to connect to the database in order to read the database’s schema. The await_database configuration allows you to control how Sequin attempts to connect to your database during startup. This option is most relevant in development environments, where you might be provisioning Sequin and your database at the same time.

By default, Sequin will wait up to 30 seconds for the database to be ready and will retry every 3 seconds. You can customize this behavior by setting the timeout_ms and interval_ms options.

​
HTTP endpoint configuration

You can configure HTTP endpoints in three ways:

​
1. External URL

http_endpoints:
  - name: "external-endpoint"         # Required
    url: "https://api.example.com/webhook"  # Required
    headers:                          # Optional
      - key: "Content-Type"
        value: "application/json"
    encrypted_headers:                # Optional, for sensitive headers
      - key: "Authorization" 
        value: "Bearer token123"

​
2. Local development endpoint

Sequin Cloud offers tunneling to your local machine for development. Learn more about tunneling in the CLI documentation.

http_endpoints:
  - name: "local-endpoint"     # Required
    local: "true"             # Required
    path: "/webhook"          # Optional
    headers:                  # Optional
      - key: "X-Test"
        value: "test"
    encrypted_headers:        # Optional
      - key: "X-Secret"
        value: "secret123"

​
3. Webhook.site testing endpoint

http_endpoints:
  - name: "webhook-site"     # Required
    webhook.site: "true"     # Required

​
Sink configuration

A sink streams data from a table to a destination (sink). All sinks share these configuration options:

sinks:
  - name: "my-sink"                # Required, unique name for this sink
    database: "my-database"        # Required, references database name
    table: "public.users"          # Required, [schema.]table to consume
    batch_size: 1                  # Optional, messages per batch, default: 1
    status: "active"               # Optional: active or disabled, default: active
    group_column_names: ["id"]     # Optional, columns to group messages by
    transform: "my-transform"       # Optional, reference to a transform name or "none"
    actions:                       # Optional, defaults to all
      - insert
      - update
      - delete
    filters:                       # Optional, filter which rows to consume with `where` clause
      - column_name: "status"      
        operator: "="
        comparison_value: "active"
      - column_name: "metadata"    # JSONB example
        field_path: "type.name"
        operator: "="
        comparison_value: "premium"
        field_type: "string"
    destination:                   # Required, sink-specific configuration
      type: "..."                  # Required, sink type
      ...                          # Additional sink-specific options

The destination configuration varies by sink type. Below are the configurations for each sink type:

​
Webhook sink

For sending changes to HTTP endpoints:

destination:
  type: "webhook"                 # Required
  http_endpoint: "endpoint-name"  # Required, references HTTP endpoint
  http_endpoint_path: "/custom"   # Optional, path to append to endpoint URL

​
Typesense sink

For indexing documents into a Typesense collection:

destination:
  type: "typesense"                # Required
  endpoint_url: "https://your-typesense-server:8108"  # Required
  collection_name: "products"      # Required
  api_key: "your-api-key"          # Required
  batch_size: 40                   # Optional, messages per batch, default: 40, max: 10000
  timeout_seconds: 5               # Optional, seconds to wait for response, default: 5, max: 300

​
Elasticsearch sink

For indexing documents into an Elasticsearch index:

destination:
  type: "elasticsearch"            # Required
  endpoint_url: "https://your-elasticsearch-cluster:9200"  # Required
  index_name: "products"           # Required
  auth_type: "api_key"             # Optional: api_key, basic, bearer, default: api_key
  auth_value: "your-auth-key"      # Required: API key, basic auth (username:password), or bearer token
  batch_size: 100                  # Optional, messages per batch, default: 100, max: 10000

​
Sequin Stream sink

Sequin Stream is a durable, scalable, and fault-tolerant message stream that you can use with Sequin in place of additional infrastructure like Kafka or SQS.

For pulling changes via the Sequin Stream API:

destination:
  type: "sequin_stream"          # Required

​
Kafka sink

For publishing changes to Kafka topics:

destination:
  type: "kafka"                  # Required
  hosts: "localhost:9092"        # Required, comma-separated list of brokers
  topic: "my-topic"             # Required
  tls: false                    # Optional, enable TLS, default: false
  username: "kafka-user"        # Optional, for SASL authentication
  password: "kafka-pass"        # Optional, for SASL authentication
  sasl_mechanism: "plain"       # Optional: plain, scram_sha_256, scram_sha_512

​
SQS sink

For sending changes to Amazon SQS queues:

destination:
  type: "sqs"                   # Required
  queue_url: "https://sqs.us-west-2.amazonaws.com/123/MyQueue.fifo" # Required
  access_key_id: "AKIAXXXX"    # Required
  secret_access_key: "secret"   # Required

​
SNS sink

For publishing changes to AWS SNS topics:

destination:
  type: "sns"                   # Required
  topic_arn: "arn:aws:sns:us-west-2:123456789012:my-topic" # Required, full SNS Topic ARN
  region: "us-west-2"          # Optional, inferred from topic_arn if omitted
  access_key_id: "AKIAXXXX"    # Required
  secret_access_key: "secret"   # Required

The region field is optional. If not provided, Sequin will automatically infer the region from the topic_arn. FIFO topics are supported—Sequin will detect a FIFO topic automatically when the topic_arn ends with .fifo.

​
Redis sink

For publishing changes to Redis streams:

destination:
  type: "redis_stream"                 # Required
  host: "localhost"             # Required
  port: 6379                    # Required
  stream_key: "my-stream"       # Required
  database: 0                   # Optional, Redis database number, default: 0
  tls: false                    # Optional, enable TLS, default: false
  username: "redis-user"        # Optional, for authentication
  password: "redis-pass"        # Optional, for authentication

​
RabbitMQ sink

For publishing changes to RabbitMQ exchanges:

destination:
  type: "rabbitmq"              # Required
  host: "localhost"              # Required
  port: 5672                    # Required
  exchange: "my-exchange"        # Required
  username: "guest"              # Optional, default: guest
  password: "guest"              # Optional, default: guest
  virtual_host: "/"              # Optional, default: /
  tls: false                    # Optional, enable TLS, default: false

​
GCP PubSub sink

For publishing changes to Google Cloud Pub/Sub topics:

destination:
  type: "gcp_pubsub"                # Required
  project_id: "my-project"          # Required
  topic_id: "my-topic"             # Required
  credentials:                      # Required, GCP service account credentials
    type: "service_account"
    project_id: "my-project"
    private_key_id: "key123"
    private_key: "-----BEGIN PRIVATE KEY-----\nMIIE...\n-----END PRIVATE KEY-----\n"
    client_email: "my-service-account@my-project.iam.gserviceaccount.com"
    client_id: "123456789"
    auth_uri: "https://accounts.google.com/o/oauth2/auth"
    token_uri: "https://oauth2.googleapis.com/token"
    auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs"
    client_x509_cert_url: "https://www.googleapis.com/robot/v1/metadata/x509/my-service-account%40my-project.iam.gserviceaccount.com"

The GCP PubSub sink requires a service account with permissions to publish to the specified topic. The credentials field should contain the JSON key file contents for a service account with the roles/pubsub.publisher role.

Project ID must be between 6 and 30 characters, start with a letter, and contain only lowercase letters, numbers, and hyphens. Topic ID must be between 3 and 255 characters and match the pattern: [a-zA-Z][a-zA-Z0-9-_.~+%]*.

​
Sink filters

All sink types support filtering messages based on columns in your source table. The filtering syntax is consistent across sink types:

filters:
  # Comparison filter
  - column_name: "status"           
    operator: "="                   # =, !=, >, >=, <, <=
    comparison_value: "active"
  
  # List filter
  - column_name: "status"
    operator: "in"                  # in, not in
    comparison_value:
      - "active"
      - "inactive"

  # Is null filter
  - column_name: "status"
    operator: "is null"              # is null, is not null
  
  # JSONB field filter
  - column_name: "metadata"         
    field_path: "preferences.theme" # Dot notation for nested fields
    operator: "="                   # =, !=, >, >=, <, <=
    comparison_value: "dark"
    field_type: "string"           # string, number, boolean, null, list

Additioanlly, you can specify actions to consume. By default, all actions are consumed.

sinks:
  - name: "my-sink"
    actions: ["insert", "update", "delete"]
    ...

​
Sink transforms

Transforms allow you to modify the structure of messages before they are sent to a sink. You can define transforms at the top level of your configuration and reference them in your sinks.

transforms:
  - name: "my-path-transform"           # Required, unique name for this transform
    description: "Extract record"       # Optional, description of the transform
    transform:                          # Required, transform configuration
      type: "path"                      # Required, "path" or "function"
      path: "record"                    # Optional, path to extract from the message (Required if type is "path")
  - name: "my-function-transform"
    transform:
      type: "function"
      code: |-                          # Optional, Elixir code to transform the message (Required if type is "function")
        def transform(action, record, changes, metadata) do
          %{
            id: record["id"],
            action: action
          }
        end

You can reference transforms in your sinks by name:

sinks:
  - name: "my-sink"
    transform: "my-path-transform"       # Reference the transform by name (or "none" for no transform)
    ...

​
Change retention configuration

change_retentions:
  - name: "my-retention"                  # Required
    source_database: "source-db"          # Required
    source_table_schema: "public"         # Required
    source_table_name: "users"            # Required
    destination_database: "dest-db"       # Required
    destination_table_schema: "public"    # Required  
    destination_table_name: "user_events" # Required
    actions:                             # Optional, defaults to all
      - insert
      - update 
      - delete
    filters:                             # Optional
      - column_name: "status"           
        operator: "="
        comparison_value: "active"
      - column_name: "metadata"          # JSONB column example
        field_path: "type.name"
        operator: "="
        comparison_value: "premium"
        field_type: "string"

​
Environment variable substitution

Sequin supports environment variable substitution in your YAML configuration files using the following syntax:

${VARIABLE_NAME:-default_value}

This allows you to:

  • Reference environment variables in your configuration
  • Provide default values when the environment variable is not set

For example:

databases:
  - name: "production-db"
    # Note quotes around the reference in case of special characters (see below)
    hostname: "${DB_HOST:-localhost}"
    # You can alternatively use a YAML heredoc, if your env variable might contain quotes (see below)
    password: |-
      ${DB_PASSWORD:-postgres}
    database: "${DB_NAME:-app_production}"

​
Best practices for environment variables in YAML

YAML has special characters that can affect parsing if they appear unquoted in your configuration. When using environment variables that might contain special characters (like :, -, {, }, [, ], etc.), it’s recommended to quote the entire reference:

# Safe pattern for values that might contain special characters
api_key: "${API_KEY}"
# Alternatively, use a heredoc if your string contains quotes
password: |-
  ${DB_PASSWORD}

This ensures that the substituted value is treated as a string literal regardless of its content, preventing YAML parsing errors.

​
Using with dotenv

You can use tools like dotenv to load environment variables from a .env file before running Sequin commands:

# Load environment variables from .env file and apply configuration
dotenv -- sequin config apply

This is particularly useful for:

  • Managing environment-specific configurations
  • Keeping sensitive values out of your YAML files
  • Simplifying deployment across different environments

​
Previewing interpolated YAML

You can preview how your environment variables will be interpolated into your configuration before applying it:

# View the interpolated YAML with environment variables substituted
sequin config interpolate my-config.yaml

# Write the interpolated YAML to a file
sequin config interpolate my-config.yaml --output interpolated.yaml

This helps verify that your configuration will be processed correctly, especially when using default values or complex environment variable patterns.

​
Example configuration

Here’s a complete example combining multiple resources:

account:
  name: "my-account"

users:
  - email: "admin@example.com"
    password: "adminpass123"

databases:
  - name: "production-db"
    hostname: "prod.db.example.com"  
    database: "app_production"
    slot_name: "sequin_slot"
    publication_name: "sequin_pub"

http_endpoints:
  - name: "webhook-endpoint"
    url: "https://api.example.com/webhook"
    encrypted_headers:
      - key: "Authorization"
        value: "Bearer token123"

sinks:
  - name: "user-changes"
    database: "production-db"
    table: "users"
    destination:
      type: "webhook"
      http_endpoint: "webhook-endpoint"

change_retentions:
  - name: "user-audit"
    source_database: "production-db" 
    source_table_schema: "public"
    source_table_name: "users"
    destination_database: "production-db"
    destination_table_schema: "audit"
    destination_table_name: "user_events"

​
YAML Anchors

YAML anchors allow you to reuse YAML configuration across multiple resources. For instance, you may want to create a sink for each table in a database. You can use anchors to avoid duplicating the sink configuration for each table.

# This database has three tables we want to sink from
databases:
  - name: "production-db"

# Create a re-usable sink template- in this case a GCP PubSub topic
gcp_pubsub_sink_template: &gcp_pubsub_sink_template
  destination:
    type: gcp_pubsub
    project_id: my-project-id
    topic_id: my-topic
    credentials:
      type: service_account
      private_key_id: "key123"
      ...
  database: "production-db"

# Use the sink template to create a sink for each table
sinks:
  - <<: *gcp_pubsub_sink_template
    name: "user-changes"
    table: "users"
  - <<: *gcp_pubsub_sink_template
    name: "account-changes"
    table: "accounts"
  - <<: *gcp_pubsub_sink_template
    name: "document-changes"
    table: "documents"

This setup will fan-in changes from all three tables into a single sink destination, all using the same YAML block for each table.