​
Overview

Configure Sequin resources like databases, sinks, and HTTP endpoints using YAML.

You can provide YAML configuration to Sequin in three ways:

  1. Via a configuration file using the CONFIG_FILE_PATH environment variable
  2. Directly as base64-encoded YAML using the CONFIG_FILE_YAML environment variable
  3. Via the Sequin CLI using the sequin config export|plan|apply command group

​
Schema

​
Account configuration

account:
  name: "account-name"  # Required for self-hosted deployments

​
User configuration

users:
  - email: "user@example.com"    # Required
    password: "userpassword123"  # Required

​
Database configuration

databases:
  - name: "my-database"          # Required, unique identifier
    username: "postgres"         # Default: postgres
    password: "postgres"         # Default: postgres
    hostname: "localhost"        # Required
    port: 5432                  # Default: 5432
    database: "my_database"     # Required
    pool_size: 10               # Default: 3
    slot_name: "sequin_slot"    # Required
    publication_name: "sequin_pub" # Required
    tables:                     # Optional, list of tables to track
      - table_name: "users"     # Required
        table_schema: "public"  # Default: public
        sort_column_name: "updated_at" # Required

​
HTTP endpoint configuration

You can configure HTTP endpoints in three ways:

​
1. External URL

http_endpoints:
  - name: "external-endpoint"         # Required
    url: "https://api.example.com/webhook"  # Required
    headers:                          # Optional
      - key: "Content-Type"
        value: "application/json"
    encrypted_headers:                # Optional, for sensitive headers
      - key: "Authorization" 
        value: "Bearer token123"

​
2. Local development endpoint

Sequin Cloud offers tunneling to your local machine for development. Learn more about tunneling in the CLI documentation.

http_endpoints:
  - name: "local-endpoint"     # Required
    local: "true"             # Required
    path: "/webhook"          # Optional
    headers:                  # Optional
      - key: "X-Test"
        value: "test"
    encrypted_headers:        # Optional
      - key: "X-Secret"
        value: "secret123"

​
3. Webhook.site testing endpoint

http_endpoints:
  - name: "webhook-site"     # Required
    webhook.site: "true"     # Required

​
Sink configuration

A sink streams data from a table to a destination (sink). All sinks share these configuration options:

sinks:
  - name: "my-sink"                # Required, unique name for this sink
    database: "my-database"        # Required, references database name
    table: "public.users"          # Required, [schema.]table to consume
    batch_size: 1                  # Optional, messages per batch, default: 1
    status: "active"               # Optional: active or disabled, default: active
    group_column_names: ["id"]   # Optional, columns to group messages by
    filters:                       # Optional, filter which rows to consume with `where` clause
      - column_name: "status"      
        operator: "="
        comparison_value: "active"
      - column_name: "metadata"    # JSONB example
        field_path: "type.name"
        operator: "="
        comparison_value: "premium"
        field_type: "string"
    destination:                 # Required, sink-specific configuration
      type: "..."                # Required, sink type
      ...                        # Additional sink-specific options

The destination configuration varies by sink type. Below are the configurations for each sink type:

​
Webhook sink

For sending changes to HTTP endpoints:

destination:
  type: "webhook"                 # Required
  http_endpoint: "endpoint-name"  # Required, references HTTP endpoint
  http_endpoint_path: "/custom"   # Optional, path to append to endpoint URL

​
Sequin Stream sink

Sequin Stream is a durable, scalable, and fault-tolerant message stream that you can use with Sequin in place of additional infrastructure like Kafka or SQS.

For pulling changes via the Sequin Stream API:

destination:
  type: "sequin_stream"          # Required

​
Kafka sink

For publishing changes to Kafka topics:

destination:
  type: "kafka"                  # Required
  hosts: "localhost:9092"        # Required, comma-separated list of brokers
  topic: "my-topic"             # Required
  tls: false                    # Optional, enable TLS, default: false
  username: "kafka-user"        # Optional, for SASL authentication
  password: "kafka-pass"        # Optional, for SASL authentication
  sasl_mechanism: "plain"       # Optional: plain, scram_sha_256, scram_sha_512

​
SQS sink

For sending changes to Amazon SQS queues:

destination:
  type: "sqs"                   # Required
  queue_url: "https://sqs.us-west-2.amazonaws.com/123/MyQueue.fifo" # Required
  access_key_id: "AKIAXXXX"    # Required
  secret_access_key: "secret"   # Required

​
Redis sink

For publishing changes to Redis streams:

destination:
  type: "redis"                 # Required
  host: "localhost"             # Required
  port: 6379                    # Required
  stream_key: "my-stream"       # Required
  database: 0                   # Optional, Redis database number, default: 0
  tls: false                    # Optional, enable TLS, default: false
  username: "redis-user"        # Optional, for authentication
  password: "redis-pass"        # Optional, for authentication

​
GCP PubSub sink

For publishing changes to Google Cloud Pub/Sub topics:

destination:
  type: "gcp_pubsub"                # Required
  project_id: "my-project"          # Required
  topic_id: "my-topic"             # Required
  credentials:                      # Required, GCP service account credentials
    type: "service_account"
    project_id: "my-project"
    private_key_id: "key123"
    private_key: "-----BEGIN PRIVATE KEY-----\nMIIE...\n-----END PRIVATE KEY-----\n"
    client_email: "my-service-account@my-project.iam.gserviceaccount.com"
    client_id: "123456789"
    auth_uri: "https://accounts.google.com/o/oauth2/auth"
    token_uri: "https://oauth2.googleapis.com/token"
    auth_provider_x509_cert_url: "https://www.googleapis.com/oauth2/v1/certs"
    client_x509_cert_url: "https://www.googleapis.com/robot/v1/metadata/x509/my-service-account%40my-project.iam.gserviceaccount.com"

The GCP PubSub sink requires a service account with permissions to publish to the specified topic. The credentials field should contain the JSON key file contents for a service account with the roles/pubsub.publisher role.

Project ID must be between 6 and 30 characters, start with a letter, and contain only lowercase letters, numbers, and hyphens. Topic ID must be between 3 and 255 characters and match the pattern: [a-zA-Z][a-zA-Z0-9-_.~+%]*.

​
Sink filters

All sink types support filtering messages based on columns in your source table. The filtering syntax is consistent across sink types:

filters:
  # Comparison filter
  - column_name: "status"           
    operator: "="                   # =, !=, >, >=, <, <=
    comparison_value: "active"
  
  # List filter
  - column_name: "status"
    operator: "in"                  # in, not in
    comparison_value:
      - "active"
      - "inactive"

  # Is null filter
  - column_name: "status"
    operator: "is null"              # is null, is not null
  
  # JSONB field filter
  - column_name: "metadata"         
    field_path: "preferences.theme" # Dot notation for nested fields
    operator: "="                   # =, !=, >, >=, <, <=
    comparison_value: "dark"
    field_type: "string"           # string, number, boolean, null, list

​
Change retention configuration

change_retentions:
  - name: "my-retention"                  # Required
    source_database: "source-db"          # Required
    source_table_schema: "public"         # Required
    source_table_name: "users"            # Required
    destination_database: "dest-db"       # Required
    destination_table_schema: "public"    # Required  
    destination_table_name: "user_events" # Required
    actions:                             # Optional, defaults to all
      - insert
      - update 
      - delete
    filters:                             # Optional
      - column_name: "status"           
        operator: "="
        comparison_value: "active"
      - column_name: "metadata"          # JSONB column example
        field_path: "type.name"
        operator: "="
        comparison_value: "premium"
        field_type: "string"

​
Example configuration

Here’s a complete example combining multiple resource:

account:
  name: "my-account"

users:
  - email: "admin@example.com"
    password: "adminpass123"

databases:
  - name: "production-db"
    hostname: "prod.db.example.com"  
    database: "app_production"
    slot_name: "sequin_slot"
    publication_name: "sequin_pub"
    tables:
      - table_name: "users"
        sort_column_name: "updated_at"

http_endpoints:
  - name: "webhook-endpoint"
    url: "https://api.example.com/webhook"
    encrypted_headers:
      - key: "Authorization"
        value: "Bearer token123"

sinks:
  - name: "user-changes"
    database: "production-db"
    table: "users"
    destination:
      type: "http_push"
      http_endpoint: "webhook-endpoint"

change_retentions:
  - name: "user-audit"
    source_database: "production-db" 
    source_table_schema: "public"
    source_table_name: "users"
    destination_database: "production-db"
    destination_table_schema: "audit"
    destination_table_name: "user_events"

Was this page helpful?

Change retentionGeneral