Kafka Summit Logo
Organized by

Kafka Summit London 2019

May 13-14, 2019 | London


One Key to Rule them All

Session Level: Intermediate

At Babylon, we believe it is possible to put an accessible and affordable health service in the hands of every person on earth. One of the key components to enable this health revolution is our near real-time streaming platform. We built a streaming platform based on Kafka Streams which transform all events generated by all Babylon micro-services based on Apache Kafka, where each data transformation is handled in near real-time by Kafka Streams. Kafka keys are key to the efficient operation of Kafka and affect partitioning, ordering and indirectly the ability to alter or remove messages. In simpler domains where a simple partition key works, the process is quite simple. However, at Babylon, we now live in a post GDPR world and have the requirement such that messages must: be deletable Able to be corrected or amended. and be consumed in the correct order according to the subject of the event we find that we need a more sophisticated strategy to handle this. At Babylon, we’ve been heavily using the partition strategy functionality of Kafka, along with protobuf and a hierarchical key strategy to fulfil these requirements, along with some some unexpected benefits that we get from this approach. Key Takeaways The partition strategy is a powerful tool and should not be shy-ed away from The challenges and benefits of such an approach Sometimes seemingly crazy ideas work out well

We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.