Kafka Summit Logo
Organized by

Kafka Summit SF 2019

September 30 - October 1, 2019 | San Francisco

Building and Evolving a Dependency-Graph Based Microservice Architecture

Session Level: Intermediate

With the rising adoption of stream- and event-driven processing microservice architectures are becoming more and more complex. One challenge that many businesses face during the initial and ongoign development of these solutions is how to properly model and maintain dependencies between microservices. One specific example for this that will be used throughout this talk cleansing and enrichment of data that has been ingested into a streaming platform. For most use cases there are a lot of minor tasks that need to be performed on every piece of data before it is fully usable for processing. Some common examples are: normalize phone numbers, normalize street addresses, geocode addresses, lookup customer data and enrich record, … Most of these tasks are completely independent of each other, but some have dependencies to be run before or after other tasks – geocoding, for example, should be done only after address normalization has finished. Defining and orchestrating a complex graph of these operations is no small feat. This talk will focus on outlining the requirements and challenges that one needs to solve when trying to implement a flexible framework for solving this use case. It will then build on these requirements and present the blueprint of a generic solution and show how Kafka and Kafka Streams are a perfect fit to address and overcome most challenges. This talk, while offering some technical details is mostly targeted at people at the architecture, rather than the code, level. Listeners will gain a thorough understanding of the challenges that stream processing offers but also be provided with generic patterns that can be used to solve these challenges in their specific infrastructure.

We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.