Transforms

Transforms allow you to modify the structure of your messages before they are sent to the sink destination. This is useful for:

• Simplifying message payloads to reduce bandwidth and storage costs
• Converting from the database’s format to a format compatible with your sink destination or downstream systems
• Maintaining the same message format as you migrate from e.g. Debezium to Sequin
• Keeping payloads under size limits (e.g., SQS’ 256KB limit)

​

Path transform

Path transforms are the simplest kind of transform. Path transforms allow you to extract one field from your message using a dot-notation path. This is useful for:

1. Sending just the latest record to the destination (e.g., for materialized views in Postgres or data warehouses)
2. Extracting just the record ID to keep payloads small (e.g., for SQS with its 256KB limit)

​

Path syntax

The first part of the path is always one of the four keys of the message object: record, changes, action, or metadata.

You can optionally add additional keys to traverse the message structure. For example, record.id will extract the id field from the record object.

Some examples:

Paths can be as deep as you need them to be. For example, record.address.city will extract the city field from the address JSONB column in the record object.

​

Function transform

Function transforms allow you to write custom Elixir code to transform your messages. This is useful for more complex transformations that are not possible with the path transform, such as:

• Specifying a format that is necessary for your sink destination
• Sanitizing sensitive data, such as PII or payment card data
• Converting timestamp formats
• Adding computed fields
• And much more!

​

Function syntax

Every function transform is implemented as an Elixir transform/4 function:

def transform(action, record, changes, metadata) do
  # Your transform here
end

The transform/4 function receives each key from the message object as an argument:

• record: The full record object
• changes: The changes object
• action: The action type (insert, update, delete)
• metadata: The metadata object

​

Elixir standard library

The function transform allows you to use a subset of the Elixir standard library, including:

• String
• Map
• Enum
• Date, DateTime, and NaiveDateTime
• Kernel
• Decimal
• URI
• Base

Other parts of the Elixir standard library are not yet supported. If you need something specific, please let us know and we will be happy to help.

​

Examples

Here are some examples of how to use the function transform:

​

Extract the record ID

def transform(action, record, changes, metadata) do
  record["id"]
end

​

Format for webhook

# Format the record for ElastiCache with a specific key structure
def transform(action, record, changes, metadata) do
  record_id = record["id"]

  %{
    key: "#{metadata.table_schema}.#{metadata.table_name}.#{record_id}",
    value: record,
    ttl: 3600 # 1 hour TTL
  }
end

​

Sanitize sensitive data

# Remove or mask sensitive fields
def transform(action, record, changes, metadata) do
  record
  |> Map.drop(["password", "credit_card", "ssn"])
  |> Map.update!("email", fn email ->
    [name, domain] = String.split(email, "@")
    masked_name = String.slice(name, 0, 2) <> String.duplicate("*", String.length(name) - 2)
    masked_name <> "@" <> domain
  end)
end

​

Add computed property

# Add a computed full name field
def transform(action, record, changes, metadata) do
  first_name = record["first_name"]
  last_name = record["last_name"]
  full_name = first_name <> " " <> last_name
  name_length = String.length(full_name)

  %{
    full_name: full_name,
    name_length: name_length
  }
end

​

Convert timestamp formats

# Convert timestamp formats
def transform(action, record, changes, metadata) do
  timestamp = record["timestamp"]

  %{
    unix_timestamp: DateTime.to_unix(timestamp),
    truncated_timestamp: DateTime.truncate(timestamp, :second),
    iso8601_timestamp: DateTime.to_iso8601(timestamp)
  }
end

​

Coming soon

• Webhook transform: Send messages to a webhook for transformation before delivery

​

Testing transforms

When creating or editing a transform, Sequin will automatically capture up to 10 recent events from your database. You can see how your transform affects these events in real-time.

When changes occur in a connected database and you have the transform editor open, Sequin will capture events and display them in the editor:

This live preview helps ensure your transform will work correctly with your actual data.

​

Example use cases

​

Keeping payloads small

When using SQS (which has a 256KB payload limit), you can use a path transform to extract just the record ID:

transform:
  path: record.id

Your consumer can then query the full record details as needed.

​

Maintaining Debezium-like format

For systems expecting a Debezium-like format, use a path transform to extract just the record:

transform:
  path: record

This ensures your messages match the expected format without additional metadata.

​

Related