Transactionally start a Temporal workflow when data changes in your database
Learn how to use Sequin to trigger a Temporal workflow when data changes in your database. Ensure consistency across services and build complex, transactionally-consistent workflows.
Temporal is a workflow engine that reliably runs long-running business processes. Often, you need to guarantee consistency between business processes executed by Temporal and your database. For instance, no matter what service modifies the data in your database, you want to ensure a workflow is executed to process the change. That is, you need a transactionally-consistent workflow.
In this tutorial, you’ll learn how to transactionally trigger a Temporal workflow using Sequin. No need to hand rebuild an outbox pattern or pull in complex dependencies like Kafka or RabbitMQ.
As a common example, you’ll trigger a workflow to ensure a user is deleted across multiple services as soon as the user is removed from the database. Sequin will give you a set of guarantees that ensure the workflow is always triggered, then Temporal will give you a set of guarantees that ensure the workflow executes.
Prerequisites
You’ll create a Sequin Webhook Sink that’ll deliver a webhook to a Node.js worker whenever a user is deleted in your database. You’ll need the following:
- Sequin installed and running locally
- A Postgres database connected to Sequin
- A working understanding of Temporal’s TypeScript SDK - includingWorkflows and Workers
- Node.js installed on your machine (if you want to run the code examples)
Setup your database tables
Start by creating a mock users table. In the Postgres database connected to Sequin, run the following SQL:
Setup a Webhook Sink in Sequin
Now, you’ll setup a Sequin Webhook Sink that’ll deliver a webhook to a Node.js worker whenever a user is deleted in your database.
Create a new webhook sink
In the Sequin console, navigate to the Sink page and click “Create Sink”. Then, select “Webhook” as the sink type.
Select the users table
Select the users
table you created in the previous step as the source table.
Filter for the delete operation
In the “Filter” section, add a filter to ensure the webhook is only delivered when a user is deleted by turning on the “Delete” filter, and turning off the “Insert” and “Update” filters.
Configure the HTTP endpoint
Skip down to the “HTTP Endpoint” section, click “New HTTP Endpoint” and in the new window, name the endpoint (e.g. temporal-worker
) and enter the following url:
No need to add any headers or authentication right now. Then, go back to your Webhook Sink form, refresh the list of endpoints, and select the temporal-worker
endpoint you just created. As a last step, add the following path to the endpoint:
The fully qualified endpoint is now: http://localhost:3333/delete-user
Save the webhook sink
Give your webhook sink a name (e.g. temporal-webhook-sink
) and click “Save” to save the webhook sink.
Sequin is now watching for deletes on the users
table and will deliver a webhook to http://localhost:3333/delete-user
whenever a user is deleted.
Scaffold a TypeScript Temporal Worker
With Sequin setup and ready to deliver webhooks, you can now scaffold a TypeScript Temporal Worker.
Temporal provides a handy CLI tool to scaffold a new worker project. Run the following command to create a new worker project:
This will create a new directory called sequin-cdc-to-temporal
with a basic Temporal worker project:
Add a HTTP server to this project by adding a src/server.ts
file and installing the express package:
Now, in a terminal start the Temporal server:
The Temporal server is now running and your project is ready for a workflow and worker to be added.
Create the Temporal Workflow
In this scenario, when you delete a user, you want to trigger a Temporal workflow to delete the user across multiple services and then send the user a confirmation email. In temporal you’ll define 3 things to achieve this:
- Activities that define the individual tasks to be performed.
- A workflow that orchestrates the activities.
- A worker that will run the Temporal workflow when the Temporal task queue receives a task.
Finally, you’ll define the HTTP server to receive the webhook from Sequin and send the signal to the Temporal server to start the workflow.
Define the delete user activities
In Temporal, activities are the individual tasks that make up a workflow. Define two activities in the src/activities.ts
file with the following code:
The first cleanUpExternalSystems
activity simulates the cleanup of a user’s data in external systems by delaying for 1 second. The second sendDeletionConfirmation
activity simulates sending a confirmation email by delaying for 500 milliseconds.
Of course, in a real application, these activities would be replaced with the actual logic to clean up the user’s data and send the confirmation email. The power of Temporal is that even if these activities take hours, fail, or are interrupted, Temporal will ensure the workflow is retried and completed.
Orchestrate the activities in a workflow
Now, you’ll orchestrate the activities in a workflow. Define a workflow in the src/workflows.ts
file with the following code:
Here, you are defining a long-lived workflow that does nothing until it receives a userDeletedSignal
(that’ll be triggered by the webhook from Sequin). It then runs the two activities and completes. Importantly, the workflow also defines how to handle and retry failed activities - in this case, the workflow will retry the activities up to 5 times.
Using a long-lived workflow triggered by a signal builds on the suggested architecture from Temporal. You could just as easily build a workflow that starts each time a webhook is received and completes after the activities are run.
Define the worker to run the workflow
With your workflow defined, setup a worker to listen to the Temporal task queue and run the workflow.
First, in the src/shared.ts
file, define a constant for the task queue name:
Now, in the src/worker.ts
file, define the worker:
This worker will poll the Temporal server for tasks and execute the workflow when a task is received.
Define the HTTP server to receive the webhook
Finally, define the HTTP server to receive the webhook from Sequin and send the signal to the workflow when a user is deleted. In the src/server.ts
file, define the server with the following code:
On boot, the server starts a Temporal client connection. It then defines the /delete-user
endpoint that Sequin will deliver the webhook to. When the endpoint is called, it triggers the Temporal workflow with a signalWithStart
call.
Your temporal workflow is now defined and will be trigged whenever a webhook is delivered from Sequin.
Run the application
Now, you’re ready to run the application. Make sure the Temporal servers is still running:
In a new terminal, start the Temporal worker:
And in another terminal, start the HTTP server:
Now, delete a user from the database:
Tracing the chain of events:
Sequin detects delete
Sequin captures the delete event from the database and delivers a webhook to the HTTP server:
Webhook received
The HTTP server triggers the Temporal workflow with a signalWithStart
call:
Temporal workflow starts
The Temporal workflow starts and runs the cleanUpExternalSystems
and sendDeletionConfirmation
activities:
Temporal workflow completes
The activities complete and the workflow completes. You can see the workflow complete in the Temporal UI:
Every time a user is deleted in the database - no matter what service performs the delete - a Temporal workflow is triggered to delete the user across multiple services and then send the user a confirmation email.
Resources
The combination of Sequin and Temporal provides a powerful way to build complex, transactionally-consistent workflow. Together, you can be sure that changes in your database are always reflected across your application and services.
To build on this example, dig into the following resources:
Connect your production database
Learn how to connect your production database to Sequin.
Deploy Sequin in production
Learn how to deploy Sequin in production.
Webhook sinks
Learn more about how to configure your webhook sinks to deliver messages to your users.
Configuration reference
Learn about all the configuration options available in sequin.yaml.
Monitor your workflows
Learn how to monitor your system with Sequin’s built-in metrics.