Kafka has become a critical part of most large-scale distributed infrastructures. We provide it as a self-service tool where topics are first-class citizens. Due to its nature, Kafka is used as a connector between projects of different teams. This is one thing that makes it challenging to provide such a service for operations, multi-tenancy and security in large organizations.
In this talk we will present how we built infrastructure that enables teams to use Kafka while keeping things secure and flexible, and while keeping the operational overload low. We’ll talk about our data model, cluster discovery, our end-to-end asymmetric encryption framework and identity management.