Get started with Kafka

In this quickstart, you’ll create a real-time data pipeline that streams changes from a Postgres database to a Kafka topic. You’ll:

Boot Sequin
Connect to a sample playground database
Configure a Kafka topic to receive database changes
See your changes flow in real-time

By the end, you’ll have hands-on experience setting up Postgres change data capture (CDC) with Sequin and Kafka.

This is the quickstart for streaming Postgres to Kafka. See the how-to guide for an explanation of how to use the Kafka sink or the reference for details on all configuration options.

Boot Kafka (optional)

If you don’t already have Kafka running, start Kafka with Docker Compose:

Download our Docker compose file for Kafka (right click, save link as…).
Move it to a new directory, navigate to that directory, and start the services:

mkdir kafka-quickstart && cd kafka-quickstart
# On macOS:
mv ~/Downloads/kafka-docker-compose.yaml docker-compose.yaml
docker compose up

Alternative: Download with curl

mkdir kafka-quickstart && cd kafka-quickstart
curl -o docker-compose.yaml https://raw.githubusercontent.com/sequinstream/sequin/main/docs/static/kafka-docker-compose.yaml
docker compose up

Alternative: Copy/paste raw Kafka docker-compose.yaml

name: playground-kafka
version: '3'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://host.docker.internal:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

If you’re using another Kafka instance, ensure you have the connection details ready.

Create the Kafka topic

Next, create the Kafka topic that will receive our database changes. Create this using the Kafka container you just started:

docker exec playground-kafka-kafka-1 \
  kafka-topics --create \
  --topic products \
  --bootstrap-server localhost:9092 \
  --partitions 1 \
  --replication-factor 1

You should see output confirming the topic was created:

Created topic products.

Run Sequin

The easiest way to get started with Sequin is with our Docker Compose file. This file starts a Postgres database, Redis instance, and Sequin server.

Create directory and start services

Download sequin-docker-compose.zip.
Unzip the file.
Navigate to the unzipped directory and start the services:

cd sequin-docker-compose && docker compose up -d

Alternative: Download with curl

curl -L https://github.com/sequinstream/sequin/releases/latest/download/sequin-docker-compose.zip -o sequin-docker-compose.zip \
  && unzip sequin-docker-compose.zip && rm sequin-docker-compose.zip
cd sequin-docker-compose && docker compose up -d

Alternative: Clone the repository

git clone git@github.com:sequinstream/sequin.git
cd sequin/docker
docker compose up -d

Verify services are running

Check that Sequin is running using docker ps:

docker ps

You should see output like the following:

CONTAINER ID   IMAGE                           COMMAND                  CREATED          STATUS                    PORTS                              NAMES
bd5c458cabde   sequin/sequin:latest            "/scripts/start_comm…"   11 seconds ago   Up 9 seconds              4000/tcp, 0.0.0.0:7376->7376/tcp   sequin-sequin-1
3bacd89765e7   grafana/grafana                 "/run.sh"                11 seconds ago   Up 11 seconds             0.0.0.0:3000->3000/tcp             sequin-sequin_grafana-1
3ad41319a66c   postgres:16                     "docker-entrypoint.s…"   11 seconds ago   Up 11 seconds (healthy)   0.0.0.0:7377->5432/tcp             sequin-sequin_postgres-1
6139a5fc4e80   redis:7                         "docker-entrypoint.s…"   11 seconds ago   Up 11 seconds             0.0.0.0:7378->6379/tcp             sequin-sequin_redis-1
7e07a5b052de   prom/prometheus                 "/bin/prometheus --c…"   11 seconds ago   Up 11 seconds             0.0.0.0:9090->9090/tcp             sequin-sequin_prometheus-1

Sequin, Postgres, Redis, Prometheus, and Grafana should be up and running (status: Up).

Login

The Docker Compose file automatically configures Sequin with an admin user and a playground database.Let’s log in to the Sequin web console:

Open the web console

After starting the Docker Compose services, open the Sequin web console at http://localhost:7376:

Sequin login page, allowing login with default credentials

Use the following default credentials to login:

Email:

admin@sequinstream.com

Password:

sequinpassword!

View the playground database

To get you started quickly, Sequin’s Docker Compose file creates a logical database called sequin_playground with a sample dataset in the public.products table.Let’s take a look:

Navigate to Databases

In the Sequin web console, click Databases in the sidebar.

Select playground database

Click on the pre-configured sequin-playground database:

The database “Health” should be green.

View contents of the products table

Let’s get a sense of what’s in the products table. Run the following command:

docker exec -i sequin-sequin_postgres-1 \
  psql -U postgres -d sequin_playground -c \
  "select id, name, price from products;"

This command connects to the running Postgres container and runs a psql command.

You should see a list of the rows in the products table:

  id |         name          | price 
----+-----------------------+-------
  1 | Avocados (3 pack)     |  5.99
  2 | Flank Steak (1 lb)    |  8.99
  3 | Salmon Fillet (12 oz) | 14.99
  4 | Baby Spinach (16 oz)  |  4.99
  5 | Sourdough Bread       |  6.99
  6 | Blueberries (6 oz)    |  3.99
(6 rows)

We’ll make modifications to this table in a bit.

Create a Kafka Sink

With the playground database connected, you can create a sink. This sink will send changes to the products table to your Kafka topic:

Navigate to Sinks

Click “Sinks” in the sidebar navigation, then click “Create Sink”.

Select sink type

Select “Kafka” as the sink type and click “Continue”.

Note "Source" configuration

In the “Source” card, note that the sequin-playground database is selected and all schemas and tables are included. Leave these defaults:

Source card showing the sequin_playground database and products table

Setup a backfill

In the Inital backfill card, select the public.products table to initate a backfill when the sink is created.

Initial backfill card with public.products table selected

Configure "Kafka Configuration"

In the “Kafka Configuration” card, enter your Kafka connection details:

Hosts: If running locally with the provided docker-compose, use host.docker.internal:9092
Topic: The Kafka topic to stream to (e.g., products)
SASL Mechanism: Select if your Kafka cluster requires authentication
Username/Password: Required if SASL is enabled
TLS: Toggle on if your Kafka cluster requires TLS

Test the connection

At the bottom of the form, click the “Test Connection” button. If you provided proper credentials, it should succeed.

Sequin can connect to your Kafka cluster.

Create the sink

You can leave the rest of the defaults. As configured, the Kafka topic will first receive a backfill of all rows currently in the products table. Then, it will receive all changes to the products table in real-time.Click “Create Sink” to finish setting up your Kafka sink.

See changes flow to your Kafka topic

On the new sink’s overview page, you should see the “Health” status turn green, indicating data is flowing to your Kafka topic.Let’s confirm messages are flowing:

Messages tab

Click the “Messages” tab. You’ll see a list of the recently delivered messages:

Sequin indicates it backfilled the products table to your Kafka topic.

View in Kafka CLI

In your terminal, run the following command to read from the topic:

docker exec playground-kafka-kafka-1 \
  kafka-console-consumer --bootstrap-server localhost:9092 \
  --topic products \
  --from-beginning \
  --max-messages 10

You should see the messages that were sent from Sequin. These are read events from the initial backfill of the products table.

Messages are flowing from Sequin to your Kafka topic.

Make some changes

Let’s make some changes to the products table and see them flow to your topic.In your terminal, run the following command to insert a new row into the products table:

docker exec -i sequin-sequin_postgres-1 \
  psql -U postgres -d sequin_playground -c \
  "insert into products (name, price) values ('Organic Honey (16 oz)', 12.99);"

In another terminal, consume messages from your Kafka topic:

docker exec playground-kafka-kafka-1 \
  kafka-console-consumer --bootstrap-server localhost:9092 \
  --topic products \
  --from-beginning

You should see a message corresponding to the inserted row.Feel free to try other changes:

Update a product's price

docker exec -i sequin-sequin_postgres-1 \
  psql -U postgres -d sequin_playground -c \
  "update products set price = 7.99 where name = 'Avocados (3 pack)';"

Change a product's name

docker exec -i sequin-sequin_postgres-1 \
  psql -U postgres -d sequin_playground -c \
  "update products set name = 'Organic Avocados (3 pack)' where name = 'Avocados (3 pack)';"

Delete a product

docker exec -i sequin-sequin_postgres-1 \
  psql -U postgres -d sequin_playground -c \
  "delete from products where name = 'Blueberries (6 oz)';"

Each change will appear in your Kafka topic within a few seconds.

Great work!

You’ve successfully:

Set up a complete Postgres change data capture pipeline
Loaded existing data through a backfill
Made changes to the products table
Verified changes are flowing to your Kafka topic

Ready to stream

Now you’re ready to connect your own database to Sequin and start streaming changes:

Guide: Connect Postgres

Connect your Postgres database to Sequin.

Guide: Setting up a Kafka sink

Setup a Kafka sink to send changes to your topic.

Boot Kafka (optional)

Create the Kafka topic

Run Sequin

Login

View the playground database

Create a Kafka Sink

See changes flow to your Kafka topic

​Ready to stream

Guide: Connect Postgres

Guide: Setting up a Kafka sink

Ready to stream