Kafka Write Mode - Dittofeed

The kafka write mode buffers data in Kafka before writing it to ClickHouse. This mode is useful when you have a high volume of data and want to avoid overwhelming ClickHouse. This mode takes advantage of ClickHouse’s kafka table engine.

Setup

To set up Dittofeed with the kafka write mode, you can make use of the following environment variables:

# necessary to use the write mode
WRITE_MODE=kafka
# the kafka host and portnames to write to
KAFKA_BROKERS=host1:port1,host2:port2
# the username and password to use for the kafka connection
KAFKA_USERNAME=my-kafka-username
KAFKA_PASSWORD=my-kafka-password
# whether to use SSL for the kafka connection
KAFKA_SSL=true
# the kafka sasl mechanism to use
# possible values are: plain, scram-sha-256, scram-sha-512
KAFKA_SASL_MECHANISM=plain
# the number of partitions to use for the user events topic when bootstrapping
KAFKA_USER_EVENTS_PARTITIONS=1
# the replication factor to use for the user events topic when bootstrapping
KAFKA_USER_EVENTS_REPLICATION_FACTOR=1
# the name of the user events topic to create when bootstrapping
USER_EVENTS_TOPIC_NAME=dittofeed-user-events

ClickHouse Kafka SASL Configuration

When using Kafka with SASL authentication, ClickHouse requires additional configuration in its config file. Create or modify the ClickHouse configuration file to include SASL settings:

<!-- clickhouse_config.xml -->
<clickhouse>
    <kafka>
        <security_protocol>sasl_plaintext</security_protocol>
        <sasl_mechanism>PLAIN</sasl_mechanism>
        <sasl_username>your-kafka-username</sasl_username>
        <sasl_password>your-kafka-password</sasl_password>
    </kafka>
</clickhouse>

For SSL-enabled Kafka connections, use:

<!-- clickhouse_config.xml -->
<clickhouse>
    <kafka>
        <security_protocol>sasl_ssl</security_protocol>
        <sasl_mechanism>PLAIN</sasl_mechanism>
        <sasl_username>your-kafka-username</sasl_username>
        <sasl_password>your-kafka-password</sasl_password>
    </kafka>
</clickhouse>

SASL authentication settings cannot be configured at the table level in ClickHouse and must be set in the server configuration file. ClickHouse requires a restart when SASL configuration changes.

Bootstrap

Finally, you can run the bootstrap script to create the necessary Kafka topic:

BOOTSTRAP_KAFKA=true ./admin.sh bootstrap

Write Modes Blob Storage

​Setup

​ClickHouse Kafka SASL Configuration

​Bootstrap

Setup

ClickHouse Kafka SASL Configuration

Bootstrap