GCP Pub/Sub sink

The GCP Pub/Sub sink sends messages to a Google Cloud Pub/Sub topic.

​

Configuration

• Project ID

  Your Google Cloud project ID (e.g., my-project-123).

• Topic ID

  The ID of your Pub/Sub topic (e.g., postgres-changes).

• Service Account Credentials

  The JSON key file contents for a service account with permissions to publish to the specified topic.

​

Service account permissions

Sequin requires a service account with the following roles:

• roles/pubsub.publisher: Allows publishing messages to topics
• roles/pubsub.viewer: Allows viewing topic configuration and metadata

For more information on creating a service account, see the how-to guide.

​

Message format

Sequin sends messages to Pub/Sub as JSON. The base message format follows the messages reference, with additional Pub/Sub specific attributes.

​

Message Structure

Each message published to Pub/Sub contains:

1. Data - The main message payload follows the messages reference structure. The data is encoded as JSON then base64 encoded.

2. Attributes - Message attributes that can be used for filtering and routing:

   • trace_id - A unique identifier for tracing the message through the system
   • type - Always “event”
   • table_name - The name of the source table
   • action - The type of change: “insert”, “update”, “delete”, or “read”

3. Ordering Key - Used for message ordering when enabled on the subscription. The key is generated based on:

   • By default, the primary key(s) of the source row
   • If custom grouping is configured, the specified grouping column values

​

Retry behavior

If Sequin is unable to deliver a message to Pub/Sub, it will retry the message indefinitely. Sequin will exponentially back off the retry interval, with a maximum backoff of roughly 3 minutes.

​

Message size limits

Pub/Sub has a maximum message size of 10MB. If Pub/Sub rejects a message due to size limitations, Sequin will cancel the message’s delivery.

​

Grouping and ordering

When you create a subscription in Pub/Sub, you can enable “message ordering” on the subscription. When enabled, messages in Pub/Sub tagged with the same ordering key will be received in the order they are published.

When you configure a Pub/Sub sink in Sequin, you can specify message grouping behavior. This will be used to set the ordering_key on the messages published to Pub/Sub.

The default message group for a message is the source row’s primary key(s). You can override this by specifying one or more columns to use for message grouping.

Sequin will order the delivery of messages with the same group according to their commit timestamp.

​

Throughput and batching

By default, Sequin publishes messages to Pub/Sub one at a time (batch_size=1). This is because the default ordering key is set to the primary key(s) of the source row, and Pub/Sub does not allow batching messages with different ordering keys.

If you don’t need message ordering in Pub/Sub, you can significantly improve throughput by:

1. Disabling message grouping - Set message_grouping: false at the sink level
2. Increasing batch size - Set batch_size to a higher value (e.g., 100-1000)

Here’s an example configuration for high throughput:

sinks:
  - name: my_pubsub_sink
    message_grouping: false  # Disables message grouping and ordering key
    batch_size: 300
    tables:
      - name: public.my_table
        # group_column_names not allowed when message_grouping is false

When message_grouping is disabled, Sequin will not set an ordering_key on the published messages, allowing Pub/Sub to process them in parallel for maximum throughput.

​

Debugging

You can view the status of your Pub/Sub sink in the Sequin web console.

On the “Messages” tab, you can see which messages are in-flight to Pub/Sub, which messages Sequin is unable to deliver, and recently delivered messages.

Messages that Sequin is unable to deliver will have a “Deliver count” greater than 1. You can click on a message to see more details, including the last error response received from Pub/Sub.

Common issues to check when debugging:

• Service account credentials are valid and not expired
• Service account has the required Pub/Sub roles
• Project ID and Topic ID match exactly
• Topic exists in the specified project

​

Local development with the emulator

Sequin supports using the GCP Pub/Sub emulator for local development. The emulator provides a local Pub/Sub environment that mimics the behavior of the real service.

To use the emulator, first consult GCP’s installation instructions.

With the emulator installed, you can start it like this:

gcloud beta emulators pubsub start --project=my-project

You can create new topics using curl:

curl -X PUT \
     -H "Content-Type: application/json" \
     http://localhost:8085/v1/projects/my-project/topics/my-topic

When setting up the sink in the Sequin console, toggle on “Use emulator” to use the emulator. For the host, use the fully-qualified URL of the emulator, e.g. http://localhost:8085.

​

Routing

The GCP PubSub sink supports dynamic routing of the topic_id with routing functions.

Example routing function:

def route(action, record, changes, metadata) do
  %{
    topic_id: "#{metadata.database_name}-#{metadata.table_schema}-#{metadata.table_name}"
  }
end

When not using a routing function, messages will be published to the topic specified in the sink configuration.