Kubernetes is fast becoming the platform of choice for running distributed, containerized applications in the cloud. It has great features for availability, scalability, monitoring, ease of deployment, a rich set of tools and an extremely fast-growing ecosystem that is making it ever more useful. However, running stateful applications such as Kafka on Kubernetes is not a common practice today. At Intuit, we took an experimentation and data-driven approach for evaluating Kafka on Kubernetes in AWS.
In this talk, we will provide details of our functional and non-functional requirements, the experimental configuration and the details of the evaluation. The evaluation process included functional tests for producing/consuming messages, network isolation tests, cross-region tests as well as performance and stress tests. We will focus on the problems we ran into and how we addressed them. This talk will demonstrate a Kubernetes cluster running Kafka along with the details of how each component is configured. Specifically, we will cover the Kafka and ZooKeeper StatefulSets, the ConfigMaps used for storing the server.properties used by all brokers, the service objects for enabling access to the brokers, securing the data and, last but not least, integration with Splunk and Wavefront for logging and monitoring respectively.