This is the reference for the Elasticsearch sink. See the quickstart for a step-by-step walkthrough or the how-to guide for an explanation of how to use the Elasticsearch sink.
Configuration
-
Endpoint URL
Base URL of your Elasticsearch cluster (for example,
https://your-es-server:9200
). - Index name Name of the Elasticsearch index to write to. The index must exist before Sequin can stream data.
-
Authentication type
One of
api_key
,basic
, orbearer
. - Authentication value Credential corresponding to the chosen authentication type.
-
Batch size (optional)
Maximum number of documents per bulk request (default
100
, maximum10 000
).
Authentication header formats
Auth type | Example header |
---|---|
api_key | Authorization: ApiKey <encoded-key> |
basic | Authorization: Basic <username:password> |
bearer | Authorization: Bearer <token> |
Transform requirements
Your transform must return a JSON document compatible with the target index’s mapping. Sequin sets the Elasticsearch document_id
to the concatenation of the primary‑key column values of the Postgres row. Let us know if you need to use a different _id
field.
Because the sink always uses the index
bulk operation, every payload must contain the full document. Partial updates are not supported.
Example transforms
Index the full document:Bulk import behaviour
For each batch Sequin posts a request to the Elasticsearch Bulk API:POST /{index_name}/_bulk
Change action | Bulk operation | Behaviour |
---|---|---|
INSERT , UPDATE , READ | index | Creates the document if missing; replaces it otherwise. |
DELETE | delete | Removes the document by _id . |
API endpoints used by Sequin
POST /{index_name}/_bulk
– indexing, replacing, and deleting documents (always used, even for a single document).POST /{index_name}/_search
– invoked by Test Connection in the console to verify connectivity.
Error handling
Typical errors surfaced in the Sequin console include:- Connection or TLS failures.
404
– index not found.401
/403
– authentication or authorisation failure.- Mapping conflicts (field type mismatches).
- Malformed bulk payload (usually due to transform output).
Limits
- Batch size is limited to
10,000
documents by Sequin. Elasticsearch’s defaulthttp.max_content_length
(100 MB) may require smaller batches. - One sink targets one index; multiple indices require multiple sinks.
- Only
index
anddelete
bulk operations are issued. Other operations (update
,create
, etc.) are not used.
Routing
The Elasticsearch sink supports dynamic routing of theindex_name
with routing functions.
Example routing function: