The GCP Pub/Sub sink sends messages to a Google Cloud Pub/Sub topic.
This is the reference for the GCP Pub/Sub sink. See the quickstart for a step-by-step walkthrough or the how-to guide for an explanation of how to use the GCP Pub/Sub sink.

​
Configuration

  • Project ID Your Google Cloud project ID (e.g., my-project-123).
  • Topic ID The ID of your Pub/Sub topic (e.g., postgres-changes).
  • Service Account Credentials The JSON key file contents for a service account with permissions to publish to the specified topic.

​
Service account permissions

Sequin requires a service account with the following roles:
  • roles/pubsub.publisher: Allows publishing messages to topics
  • roles/pubsub.viewer: Allows viewing topic configuration and metadata
For more information on creating a service account, see the how-to guide.

​
Message format

Sequin sends messages to Pub/Sub as JSON. The base message format follows the messages reference, with additional Pub/Sub specific attributes.

​
Message Structure

Each message published to Pub/Sub contains:
  1. Data - The main message payload follows the messages reference structure. The data is encoded as JSON then base64 encoded.
  2. Attributes - Message attributes that can be used for filtering and routing:
    • trace_id - A unique identifier for tracing the message through the system
    • type - Always “event”
    • table_name - The name of the source table
    • action - The type of change: “insert”, “update”, “delete”, or “read”
  3. Ordering Key - Used for message ordering when enabled on the subscription. The key is generated based on:
    • By default, the primary key(s) of the source row
    • If custom grouping is configured, the specified grouping column values

​
Retry behavior

If Sequin is unable to deliver a message to Pub/Sub, it will retry the message indefinitely. Sequin will exponentially back off the retry interval, with a maximum backoff of roughly 3 minutes.

​
Message size limits

Pub/Sub has a maximum message size of 10MB. If Pub/Sub rejects a message due to size limitations, Sequin will cancel the message’s delivery.
If you want to see logging/alerting for this situation, please upvote the corresponding issue.

​
Grouping and ordering

When you create a subscription in Pub/Sub, you can enable “message ordering” on the subscription. When enabled, messages in Pub/Sub tagged with the same ordering key will be received in the order they are published. When you configure a Pub/Sub sink in Sequin, you can specify message grouping behavior. This will be used to set the ordering_key on the messages published to Pub/Sub. The default message group for a message is the source row’s primary key(s). You can override this by specifying one or more columns to use for message grouping. Sequin will order the delivery of messages with the same group according to their commit timestamp.

​
Throughput and batching

By default, Sequin configures GCP Pub/Sub sinks without message ordering to optimize throughput. If you need message ordering in Pub/Sub, you can enable it on the subscription and set the message_grouping option to true at the sink level.

​
Debugging

You can view the status of your Pub/Sub sink in the Sequin web console. On the “Messages” tab, you can see which messages are in-flight to Pub/Sub, which messages Sequin is unable to deliver, and recently delivered messages. Messages that Sequin is unable to deliver will have a “Deliver count” greater than 1. You can click on a message to see more details, including the last error response received from Pub/Sub. Common issues to check when debugging:
  • Service account credentials are valid and not expired
  • Service account has the required Pub/Sub roles
  • Project ID and Topic ID match exactly
  • Topic exists in the specified project

​
Local development with the emulator

Sequin supports using the GCP Pub/Sub emulator for local development. The emulator provides a local Pub/Sub environment that mimics the behavior of the real service. To use the emulator, first consult GCP’s installation instructions. With the emulator installed, you can start it like this: 
gcloud beta emulators pubsub start --project=my-project
You can create new topics using curl: 
curl -X PUT \
     -H "Content-Type: application/json" \
     http://localhost:8085/v1/projects/my-project/topics/my-topic
When setting up the sink in the Sequin console, toggle on “Use emulator” to use the emulator. For the host, use the fully-qualified URL of the emulator, e.g. http://localhost:8085.

​
Routing

The GCP PubSub sink supports dynamic routing of the topic_id with routing functions. Example routing function: 
def route(action, record, changes, metadata) do
  %{
    topic_id: "#{metadata.database_name}-#{metadata.table_schema}-#{metadata.table_name}"
  }
end
When not using a routing function, messages will be published to the topic specified in the sink configuration.