We will cover a new solution and discuss the challenges in replicating database transactions, consisting of multiple ordered operations, to Kafka and through to the end consumer application. Be it a Z mainframe, DB2, Oracle or other database, there exists a need for Kafka consumer applications to understand the transactional context and operation ordering of the original source database’s transactions.
The solution will focus on how to layer the guarantees of a relational database with respect to ACID transaction properties over Kafka and provide them to an end consumer. Challenges to be discussed include:
- How to maintain Kafka’s performance while providing greatly improved semantic guarantees
- How to address the simultaneous need to maintain topic per database table representations in Kafka and yet provide the Kafka consumer the order of operations within a transaction that apply to any number of topics
- A demonstration of how a source database’s referential integrity-constrained tables can have their operational data replicated through Kafka and ultimately applied to another database with the same constraints
- Mechanisms for consumer applications to leverage this new functionality to process consumed transactions exactly once
- How to handle outages in the middle of replication and ensure semantic guarantees are maintained from the consumer’s perspective
The talk will include a demonstration of transactionally consistent, ordered, database replication leveraging a source database, IBM IIDR CDC Replication to Message Hub in the cloud, and resultant consumer output.