​
Overview

Configure Sequin resources like databases, sinks, and HTTP endpoints using YAML.

You can provide YAML configuration to Sequin in three ways:

  1. Via a configuration file using the CONFIG_FILE_PATH environment variable
  2. Directly as base64-encoded YAML using the CONFIG_FILE_YAML environment variable
  3. Via the Sequin CLI using the sequin config export|plan|apply command group

​
Schema

​
Account configuration

account:
  name: "account-name"  # Required for self-hosted deployments

Creating accounts is only supported on self-hosted Sequin.

​
User configuration

users:
  - email: "user@example.com"    # Required
    password: "userpassword123"  # Required

​
API token configuration

You can create API tokens for your account. This is intended for use in development or CI/CD workflows:

api_tokens:
  - name: "mytoken"
    token: "secret"

Creating API tokens is only supported on self-hosted Sequin.

​
Database configuration

databases:
  - name: "my-database"          # Required, unique identifier
    username: "postgres"         # Default: postgres
    password: "postgres"         # Default: postgres
    hostname: "localhost"        # Required
    port: 5432                  # Default: 5432
    database: "my_database"     # Required
    pool_size: 10               # Default: 3
    slot_name: "sequin_slot"    # Required
    publication_name: "sequin_pub" # Required
    await_database:             # Optional, configuration for database connection retry behavior
      timeout_ms: 30000         # Default: 30000 (30 seconds)
      interval_ms: 3000         # Default: 3000 (3 seconds)
    tables:                     # Optional, list of tables to track
      - table_name: "users"     # Required
        table_schema: "public"  # Default: public

​
Database connection retry

When creating a database, Sequin needs to be able to connect to the database in order to read the database’s schema. The await_database configuration allows you to control how Sequin attempts to connect to your database during startup. This option is most relevant in development environments, where you might be provisioning Sequin and your database at the same time.

By default, Sequin will wait up to 30 seconds for the database to be ready and will retry every 3 seconds. You can customize this behavior by setting the timeout_ms and interval_ms options.

​
HTTP endpoint configuration

You can configure HTTP endpoints in three ways:

​
1. External URL

http_endpoints:
  - name: "external-endpoint"         # Required
    url: "https://api.example.com/webhook"  # Required
    headers:                          # Optional
      - key: "Content-Type"
        value: "application/json"
    encrypted_headers:                # Optional, for sensitive headers
      - key: "Authorization" 
        value: "Bearer token123"

​
2. Local development endpoint

Sequin Cloud offers tunneling to your local machine for development. Learn more about tunneling in the CLI documentation.

http_endpoints:
  - name: "local-endpoint"     # Required
    local: "true"             # Required
    path: "/webhook"          # Optional
    headers:                  # Optional
      - key: "X-Test"
        value: "test"
    encrypted_headers:        # Optional
      - key: "X-Secret"
        value: "secret123"

​
3. Webhook.site testing endpoint

http_endpoints:
  - name: "webhook-site"     # Required
    webhook.site: "true"     # Required

​
Sink configuration

A sink streams data from a table to a destination (sink). All sinks share these configuration options:

sinks:
  - name: "my-sink"                # Required, unique name for this sink
    database: "my-database"        # Required, references database name
    table: "public.users"          # Required, [schema.]table to consume
    batch_size: 1                  # Optional, messages per batch, default: 1
    status: "active"               # Optional: active or disabled, default: active
    group_column_names: ["id"]     # Optional, columns to group messages by
    transform: "my-transform"       # Optional, reference to a transform name or "none"
    actions:                       # Optional, defaults to all
      - insert
      - update
      - delete
    filters:                       # Optional, filter which rows to consume with `where` clause
      - column_name: "status"      
        operator: "="
        comparison_value: "active"
      - column_name: "metadata"    # JSONB example
        field_path: "type.name"
        operator: "="
        comparison_value: "premium"
        field_type: "string"
    destination:                   # Required, sink-specific configuration
      type: "..."                  # Required, sink type
      ...                          # Additional sink-specific options

The destination configuration varies by sink type. Below are the configurations for each sink type:

​
Webhook sink

For sending changes to HTTP endpoints:

destination:
  type: "webhook"                 # Required
  http_endpoint: "endpoint-name"  # Required, references HTTP endpoint
  http_endpoint_path: "/custom"   # Optional, path to append to endpoint URL

​
Sequin Stream sink

Sequin Stream is a durable, scalable, and fault-tolerant message stream that you can use with Sequin in place of additional infrastructure like Kafka or SQS.

For pulling changes via the Sequin Stream API:

destination:
  type: "sequin_stream"          # Required

​
Kafka sink

For publishing changes to Kafka topics:

destination:
  type: "kafka"                  # Required
  hosts: "localhost:9092"        # Required, comma-separated list of brokers
  topic: "my-topic"             # Required
  tls: false                    # Optional, enable TLS, default: false
  username: "kafka-user"        # Optional, for SASL authentication
  password: "kafka-pass"        # Optional, for SASL authentication
  sasl_mechanism: "plain"       # Optional: plain, scram_sha_256, scram_sha_512

​
SQS sink

For sending changes to Amazon SQS queues:

destination:
  type: "sqs"                   # Required
  queue_url: "https://sqs.us-west-2.amazonaws.com/123/MyQueue.fifo" # Required
  access_key_id: "AKIAXXXX"    # Required
  secret_access_key: "secret"   # Required

​
Redis sink

For publishing changes to Redis streams:

destination:
  type: "redis"                 # Required
  host: "localhost"             # Required
  port: 6379                    # Required
  stream_key: "my-stream"       # Required
  database: 0                   # Optional, Redis database number, default: 0
  tls: false                    # Optional, enable TLS, default: false
  username: "redis-user"        # Optional, for authentication
  password: "redis-pass"        # Optional, for authentication

​
RabbitMQ sink

For publishing changes to RabbitMQ exchanges:

destination:
  type: "rabbitmq"              # Required
  host: "localhost"              # Required
  port: 5672                    # Required
  exchange: "my-exchange"        # Required
  username: "guest"              # Optional, default: guest
  password: "guest"              # Optional, default: guest
  virtual_host: "/"              # Optional, default: /
  tls: false                    # Optional, enable TLS, default: false

​
GCP PubSub sink

For publishing changes to Google Cloud Pub/Sub topics:

destination:
  type: "gcp_pubsub"                # Required
  project_id: "my-project"          # Required
  topic_id: "my-topic"             # Required
  credentials:                      # Required, GCP service account credentials
    type: "service_account"
    project_id: "my-project"
    private_key_id: "key123"
    private_key: "-----BEGIN PRIVATE KEY-----\nMIIE...\n-----END PRIVATE KEY-----\n"
    client_email: "my-service-account@my-project.iam.gserviceaccount.com"
    client_id: "123456789"
    auth_uri: "https://accounts.google.com/o/oauth2/auth"
    token_uri: "https://oauth2.googleapis.com/token"
    auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs"
    client_x509_cert_url: "https://www.googleapis.com/robot/v1/metadata/x509/my-service-account%40my-project.iam.gserviceaccount.com"

The GCP PubSub sink requires a service account with permissions to publish to the specified topic. The credentials field should contain the JSON key file contents for a service account with the roles/pubsub.publisher role.

Project ID must be between 6 and 30 characters, start with a letter, and contain only lowercase letters, numbers, and hyphens. Topic ID must be between 3 and 255 characters and match the pattern: [a-zA-Z][a-zA-Z0-9-_.~+%]*.

​
Sink filters

All sink types support filtering messages based on columns in your source table. The filtering syntax is consistent across sink types:

filters:
  # Comparison filter
  - column_name: "status"           
    operator: "="                   # =, !=, >, >=, <, <=
    comparison_value: "active"
  
  # List filter
  - column_name: "status"
    operator: "in"                  # in, not in
    comparison_value:
      - "active"
      - "inactive"

  # Is null filter
  - column_name: "status"
    operator: "is null"              # is null, is not null
  
  # JSONB field filter
  - column_name: "metadata"         
    field_path: "preferences.theme" # Dot notation for nested fields
    operator: "="                   # =, !=, >, >=, <, <=
    comparison_value: "dark"
    field_type: "string"           # string, number, boolean, null, list

Additioanlly, you can specify actions to consume. By default, all actions are consumed.

sinks:
  - name: "my-sink"
    actions: ["insert", "update", "delete"]
    ...

​
Sink transforms

Transforms allow you to modify the structure of messages before they are sent to a sink. You can define transforms at the top level of your configuration and reference them in your sinks.

transforms:
  - name: "my-transform"          # Required, unique name for this transform
    transform:                    # Required, transform configuration
      type: "path"                # Required, currently only "path" is supported
      path: "record"              # Required, path to extract from the message

The path field supports the following paths:

  • record - The entire record object
  • changes - The changes object (only available for event messages)
  • action - The action type (insert, update, delete) (only available for event messages)
  • metadata - The metadata object

You can also use dot notation to access nested fields within these objects, for example:

  • record.id - The ID field from the record
  • changes.name - The name field from the changes
  • metadata.transaction_annotations.user_id - A custom transaction annotation

Example usage:

transforms:
  - name: "record-only"
    transform:
      type: "path"
      path: "record"

  - name: "id-only"
    transform:
      type: "path"
      path: "record.id"

  - name: "metadata-only"
    transform:
      type: "path"
      path: "metadata"

sinks:
  - name: "user-changes"
    database: "my-database"
    table: "users"
    transform: "record-only"      # Reference the transform by name
    destination:
      type: "webhook"
      http_endpoint: "my-endpoint"

​
Change retention configuration

change_retentions:
  - name: "my-retention"                  # Required
    source_database: "source-db"          # Required
    source_table_schema: "public"         # Required
    source_table_name: "users"            # Required
    destination_database: "dest-db"       # Required
    destination_table_schema: "public"    # Required  
    destination_table_name: "user_events" # Required
    actions:                             # Optional, defaults to all
      - insert
      - update 
      - delete
    filters:                             # Optional
      - column_name: "status"           
        operator: "="
        comparison_value: "active"
      - column_name: "metadata"          # JSONB column example
        field_path: "type.name"
        operator: "="
        comparison_value: "premium"
        field_type: "string"

​
Environment variable substitution

Sequin supports environment variable substitution in your YAML configuration files using the following syntax:

${VARIABLE_NAME:-default_value}

This allows you to:

  • Reference environment variables in your configuration
  • Provide default values when the environment variable is not set

For example:

databases:
  - name: "production-db"
    hostname: "${DB_HOST:-localhost}"
    password: "${DB_PASSWORD:-postgres}"
    database: "${DB_NAME:-app_production}"

​
Using with dotenv

You can use tools like dotenv to load environment variables from a .env file before running Sequin commands:

# Load environment variables from .env file and apply configuration
dotenv -- sequin config apply

This is particularly useful for:

  • Managing environment-specific configurations
  • Keeping sensitive values out of your YAML files
  • Simplifying deployment across different environments

​
Example configuration

Here’s a complete example combining multiple resources:

account:
  name: "my-account"

users:
  - email: "admin@example.com"
    password: "adminpass123"

databases:
  - name: "production-db"
    hostname: "prod.db.example.com"  
    database: "app_production"
    slot_name: "sequin_slot"
    publication_name: "sequin_pub"
    tables:
      - table_name: "users"

http_endpoints:
  - name: "webhook-endpoint"
    url: "https://api.example.com/webhook"
    encrypted_headers:
      - key: "Authorization"
        value: "Bearer token123"

sinks:
  - name: "user-changes"
    database: "production-db"
    table: "users"
    destination:
      type: "webhook"
      http_endpoint: "webhook-endpoint"

change_retentions:
  - name: "user-audit"
    source_database: "production-db" 
    source_table_schema: "public"
    source_table_name: "users"
    destination_database: "production-db"
    destination_table_schema: "audit"
    destination_table_name: "user_events"

​
YAML Anchors

YAML anchors allow you to reuse YAML configuration across multiple resources. For instance, you may want to create a sink for each table in a database. You can use anchors to avoid duplicating the sink configuration for each table.

# This database has three tables we want to sink from
databases:
  - name: "production-db"
    tables:
      - table_name: "users"
        ...
      - table_name: "accounts"
        ...
      - table_name: "documents"
        ...

# Create a re-usable sink template- in this case a GCP PubSub topic
gcp_pubsub_sink_template: &gcp_pubsub_sink_template
  destination:
    type: gcp_pubsub
    project_id: my-project-id
    topic_id: my-topic
    credentials:
      type: service_account
      private_key_id: "key123"
      ...
  database: "production-db"

# Use the sink template to create a sink for each table
sinks:
  - <<: *gcp_pubsub_sink_template
    name: "user-changes"
    table: "users"
  - <<: *gcp_pubsub_sink_template
    name: "account-changes"
    table: "accounts"
  - <<: *gcp_pubsub_sink_template
    name: "document-changes"
    table: "documents"

This setup will fan-in changes from all three tables into a single sink destination, all using the same YAML block for each table.