Kafka organizes data as immutable append-only logs at its core, and relied on external consensus services (a.k.a. Zookeeper) to manage the metadata --- such as topic-level configs, leader replicas and ISR information, received admin requests --- of these logs. In this talk, I will discuss a recent core initiative, that migrates the management of such metadata from external services into Kafka as its own special logs. More specifically, I will cover the following: 1. Why we believe an internal consensus protocol provides Kafka more benefit than an external consensus service. 2. Why we choose to build this internal "metadata log" based on the Raft protocol, instead of Kafka's current leader-follower replication mechanism. 3. What are the key design decisions we made in its implementation, and how it is different from the standard Raft algorithm (KIP-595). 4. How this Raft-based metadata log is leveraged by the new Quorum Controller (KIP-500).
