Kafka Summit Logo
Organized by

Kafka Summit San Francisco 2018

Streaming platforms at massive scale.

October 16-17, 2018 | San Francisco

Let’s Build ‘eBay’ by ‘Turning the Database Inside Out’ and Using Serverless

Session Level: Intermediate
Video & Slides

Apache Kafka has an immense capability for building data-intensive applications. The ability to model and scale to thousands of parallel real-time streams bring with it incredible fundamentals upon which you can build scalable systems of almost any nature. We also have the brave new world of serverless that causes much confusion as to how to use Functions as a Service (FaaS) and streaming together.

This talk builds a horizontally scalable auction platform with real-time marketplace analytics to work at the scale of eBay. The design uses the concepts of ‘turning the database inside out” to build a model of how to make a stream platform works synergistically with FaaS/AWS Lambda. First, we explore the founding principles of the log, producer/consumer topics, partitions and events. We then build up to Kafka Streams stateless and stateful stream processing and KSQL. These principles are mapped onto simple use cases in order to establish how to build higher order functionality. These use cases are combined to develop an architecture that provides the design semantics required for a real-time auction system and marketplace intelligence. The architecture is composed of a queryable data fabric using Kafka Streams state stores, a high-throughput worker-queue using exactly-once semantics (EoS) Kafka consumers, and a queue-worker hook to drive AWS Lambda functions.

The end result is a real-time system that scales elastically to service millions of auction events and provide live, marketplace analytics. The audience will learn how to compose large scale applications using all of the Apache Kafka stack as well as how we view “turning the database inside out” when used in conjunction with serverless architectures.


We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.