sequin.yaml
Reference for Sequin’s YAML configuration for infrastructure-as-code deployments.
Overview
Configure Sequin resources like databases, sinks, and HTTP endpoints using YAML.
You can provide YAML configuration to Sequin in three ways:
- Via a configuration file using the
CONFIG_FILE_PATH
environment variable - Directly as base64-encoded YAML using the
CONFIG_FILE_YAML
environment variable - Via the Sequin CLI using the
sequin config export|plan|apply
command group
Schema
Account configuration
Creating accounts is only supported on self-hosted Sequin.
User configuration
API token configuration
You can create API tokens for your account. This is intended for use in development or CI/CD workflows:
Creating API tokens is only supported on self-hosted Sequin.
Database configuration
Replica database configuration
When connecting a replica to Sequin, Sequin also needs to connect to the primary database.
Replication slot and publication configuration
Sequin uses PostgreSQL’s logical replication to capture changes from your database. This requires a replication slot and a publication.
Replication slot
After following the instructions in the Sequin console to create a replication slot, you can indicate the name of the slot in the slot
block. Alternatively, you can choose to have Sequin create the slot for you by setting create_if_not_exists
to true
:
Replication slots are durable and can accumulate data if not in use (i.e. if Sequin is not running). So while Sequin can automatically create slots for you, slots will hang around until you drop them–even if you disconnect Sequin.
Publication
After following the instructions in the Sequin console to create a publication, you can indicate the name of the publication in the publication
block. Alternatively, you can choose to have Sequin create the publication for you by setting create_if_not_exists
to true
:
When create_if_not_exists
is true
, Sequin will attempt to create the publication if it doesn’t exist. If you specify init_sql
, Sequin will run that SQL to create the publication. Otherwise, Sequin will create a publication for the schema “public”.
Database connection retry
When creating a database, Sequin needs to be able to connect to the database in order to read the database’s schema. The await_database
configuration allows you to control how Sequin attempts to connect to your database during startup. This option is most relevant in development environments, where you might be provisioning Sequin and your database at the same time.
By default, Sequin will wait up to 30 seconds for the database to be ready and will retry every 3 seconds. You can customize this behavior by setting the timeout_ms
and interval_ms
options.
Sink configuration
A sink streams data from a table to a destination (sink). All sinks share these configuration options:
The transform
, filter
, and routing
fields allow you to attach functions to your sink. Each field can be set to either:
- The name of a function defined in the
functions
section "none"
to explicitly disable that type of function
For example, to attach a transform function to your sink:
See Function configuration for more details on how to define functions.
Destination configuration
The destination
configuration varies by sink type. Below are the configurations for each sink type:
Webhook sink
For sending changes to HTTP endpoints:
Typesense sink
For indexing documents into a Typesense collection:
Elasticsearch sink
For indexing documents into an Elasticsearch index:
Sequin Stream sink
Sequin Stream is a durable, scalable, and fault-tolerant message stream that you can use with Sequin in place of additional infrastructure like Kafka or SQS.
For pulling changes via the Sequin Stream API:
Kafka sink
For publishing changes to Kafka topics:
SQS sink
For sending changes to Amazon SQS queues:
SNS sink
For publishing changes to AWS SNS topics:
The region
field is optional. If not provided, Sequin will automatically infer the region from the topic_arn
. FIFO topics are supported—Sequin will detect a FIFO topic automatically when the topic_arn
ends with .fifo
.
Redis sink
For publishing changes to Redis streams:
RabbitMQ sink
For publishing changes to RabbitMQ exchanges:
GCP PubSub sink
For publishing changes to Google Cloud Pub/Sub topics:
The GCP PubSub sink requires a service account with permissions to publish to the specified topic. The credentials
field should contain the JSON key file contents for a service account with the roles/pubsub.publisher
role.
Project ID must be between 6 and 30 characters, start with a letter, and contain only lowercase letters, numbers, and hyphens. Topic ID must be between 3 and 255 characters and match the pattern: [a-zA-Z][a-zA-Z0-9-_.~+%]*
.
Sink filters
All sink types support filtering messages based on columns in your source table. The filtering syntax is consistent across sink types:
Additioanlly, you can specify actions to consume. By default, all actions are consumed.
Function configuration
Functions allow you to filter, transform, and route your data as it flows through Sequin. You can define functions at the top level of your configuration and then attach them to your sinks.
Function types
Sequin supports four types of functions:
- Path functions - Extract data from a specific path in your message
- Transform functions - Modify the structure of your messages
- Filter functions - Filter which messages to process
- Routing functions - Dynamically direct messages to different destinations
You can define functions in your configuration like this:
Using functions in sinks
You can attach functions to your sinks using the transform
, filter
, and routing
fields:
Each field can be set to either:
- The name of a function defined in the
functions
section "none"
to explicitly disable that type of function
HTTP endpoint configuration
You can configure HTTP endpoints in three ways:
1. External URL
2. Local development endpoint
Sequin Cloud offers tunneling to your local machine for development. Learn more about tunneling in the CLI documentation.
3. Webhook.site testing endpoint
Change retention configuration
Environment variable substitution
Sequin supports environment variable substitution in your YAML configuration files using the following syntax:
This allows you to:
- Reference environment variables in your configuration
- Provide default values when the environment variable is not set
For example:
Best practices for environment variables in YAML
YAML has special characters that can affect parsing if they appear unquoted in your configuration. When using environment variables that might contain special characters (like :
, -
, {
, }
, [
, ]
, etc.), it’s recommended to quote the entire reference:
This ensures that the substituted value is treated as a string literal regardless of its content, preventing YAML parsing errors.
Using with dotenv
You can use tools like dotenv
to load environment variables from a .env
file before running Sequin commands:
This is particularly useful for:
- Managing environment-specific configurations
- Keeping sensitive values out of your YAML files
- Simplifying deployment across different environments
Previewing interpolated YAML
You can preview how your environment variables will be interpolated into your configuration before applying it:
This helps verify that your configuration will be processed correctly, especially when using default values or complex environment variable patterns.
Example configuration
Here’s a complete example combining multiple resources:
YAML Anchors
YAML anchors allow you to reuse YAML configuration across multiple resources. For instance, you may want to create a sink for each table in a database. You can use anchors to avoid duplicating the sink configuration for each table.
This setup will fan-in changes from all three tables into a single sink destination, all using the same YAML block for each table.