Debezium | Spice.ai Cloud Documentation

Debezium

Debezium is an open-source platform that enables Change Data Capture (CDC) for efficient real-time updates of locally accelerated datasets. Spice supports connecting to a Kafka topic managed by Debezium to keep datasets up-to-date with the source data.

Configuration

`from`

The from field takes the form of debezium:kafka_topic where kafka_topic is the name of the Kafka topic where Debezium is notifying consumers about any upstream changes. In the example above it would listen to the my_kafka_topic_with_debezium_changes topic.

`name`

The dataset name. This will be used as the table name within Spice.

datasets:
  - from: debezium:my_kafka_topic_with_debezium_changes
    name: cool_dataset

SELECT COUNT(*) FROM cool_dataset;

+----------+
| count(*) |
+----------+
| 6001215  |
+----------+

`params`

Parameter Name	Description
`debezium_transport`	Optional. The message broker transport to use. The default is `kafka`. Possible values: `kafka`.
`debezium_message_format`	Optional. The message format to use. The default is `json`. Possible values: `json`.
`kafka_bootstrap_servers`	Required. A list of host/port pairs for establishing the initial Kafka cluster connection.
`kafka_security_protocol`	Security protocol for Kafka connections. Default: `SASL_SSL`. Options: `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, `SASL_SSL`.
`kafka_sasl_mechanism`	SASL (Simple Authentication and Security Layer) authentication mechanism. Default: `SCRAM-SHA-512`.
`kafka_sasl_username`	SASL username.
`kafka_sasl_password`	SASL password.
`kafka_ssl_ca_location`	Path to the SSL/TLS CA certificate file for server verification.
`kafka_enable_ssl_certificate_verification`	Enable SSL/TLS certificate verification. Default: `true`.
`kafka_ssl_endpoint_identification_algorithm`	SSL/TLS endpoint identification algorithm. Default: `https`. Options: `none`, `https`.

Acceleration Settings

Using the Debezium connector requires acceleration to be enabled.

Parameter Name	Description
`enabled`	Required. Must be set to `true` to enable acceleration.
`engine`	Required. The acceleration engine to use. Valid values: `duckdb`, `sqlite`, `postgres`.
`refresh_mode`	Optional. The refresh mode to use. If specified, this must be set to `changes`. Any other value is an error.
`mode`	Optional. The persistence mode to use. Recommended to set this to `file` to persist the data across restarts.