Nobody likes dealing with duplicated orders or missing payments, so it’s crucial for the data pipelines we build to process these things to ensure each message is processed exactly once. While Kafka has supported exactly-once semantics for years in its clients and stream processing libraries, Kafka Connect lacked this support for source connectors until version 3.3. In this talk, learn how KIP-618 made exactly-once source connectors possible. Topics covered will include an overview of exactly-once support in Kafka’s client libraries, a brief refresher on the source connector API, a deep dive into some of the internal workings of Kafka Connect, and the design challenges of EOS support. This talk assumes basic familiarity with Kafka, its client libraries, and Kafka Connect. Audience members should expect to come away with better knowledge of how to implement exactly-once source connectors and how to run Kafka Connect clusters with exactly-once support.
