Kafka Summit Logo
Organized by

Kafka Summit San Francisco 2018

Streaming platforms at massive scale.

October 16-17, 2018 | San Francisco

Transactionally Consistent, Ordered, Exactly-Once Replication from the Database to Kafka in the Cloud and Back—A Solution for Leveraging Kafka to Provide End-to-End ACID Transactions

Video & Slides

We will cover a new solution and discuss the challenges in replicating database transactions, consisting of multiple ordered operations, to Kafka and through to the end consumer application. Be it a Z mainframe, DB2, Oracle or other database, there exists a need for Kafka consumer applications to understand the transactional context and operation ordering of the original source database’s transactions.

The solution will focus on how to layer the guarantees of a relational database with respect to ACID transaction properties over Kafka and provide them to an end consumer. Challenges to be discussed include:

  • How to maintain Kafka’s performance while providing greatly improved semantic guarantees
  • How to address the simultaneous need to maintain topic per database table representations in Kafka and yet provide the Kafka consumer the order of operations within a transaction that apply to any number of topics
  • A demonstration of how a source database’s referential integrity-constrained tables can have their operational data replicated through Kafka and ultimately applied to another database with the same constraints
  • Mechanisms for consumer applications to leverage this new functionality to process consumed transactions exactly once
  • How to handle outages in the middle of replication and ensure semantic guarantees are maintained from the consumer’s perspective

The talk will include a demonstration of transactionally consistent, ordered, database replication leveraging a source database, IBM IIDR CDC Replication to Message Hub in the cloud, and resultant consumer output.


We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.