Reference for configuring and using the Google Cloud Pub/Sub sink with Sequin.
my-project-123
).
postgres-changes
).
roles/pubsub.publisher
: Allows publishing messages to topicsroles/pubsub.viewer
: Allows viewing topic configuration and metadatatrace_id
- A unique identifier for tracing the message through the systemtype
- Always “event”table_name
- The name of the source tableaction
- The type of change: “insert”, “update”, “delete”, or “read”ordering_key
on the messages published to Pub/Sub.
The default message group for a message is the source row’s primary key(s). You can override this by specifying one or more columns to use for message grouping.
Sequin will order the delivery of messages with the same group according to their commit timestamp.
batch_size=1
). This is because the default ordering key is set to the primary key(s) of the source row, and Pub/Sub does not allow batching messages with different ordering keys.
If you don’t need message ordering in Pub/Sub, you can significantly improve throughput by:
message_grouping: false
at the sink levelbatch_size
to a higher value (e.g., 100-1000)message_grouping
is disabled, Sequin will not set an ordering_key
on the published messages, allowing Pub/Sub to process them in parallel for maximum throughput.
1
. You can click on a message to see more details, including the last error response received from Pub/Sub.
Common issues to check when debugging:
curl
:
http://localhost:8085
.
topic_id
with routing functions.
Example routing function: