Apache Kafka has an immense capability for building data-intensive applications. The ability to model and scale to thousands of parallel real-time streams bring with it incredible fundamentals upon which you can build scalable systems of almost any nature. We also have the brave new world of serverless that causes much confusion as to how to use Functions as a Service (FaaS) and streaming together.
This talk builds a horizontally scalable auction platform with real-time marketplace analytics to work at the scale of eBay. The design uses the concepts of ‘turning the database inside out” to build a model of how to make a stream platform works synergistically with FaaS/AWS Lambda. First, we explore the founding principles of the log, producer/consumer topics, partitions and events. We then build up to Kafka Streams stateless and stateful stream processing and KSQL. These principles are mapped onto simple use cases in order to establish how to build higher order functionality. These use cases are combined to develop an architecture that provides the design semantics required for a real-time auction system and marketplace intelligence. The architecture is composed of a queryable data fabric using Kafka Streams state stores, a high-throughput worker-queue using exactly-once semantics (EoS) Kafka consumers, and a queue-worker hook to drive AWS Lambda functions.
The end result is a real-time system that scales elastically to service millions of auction events and provide live, marketplace analytics. The audience will learn how to compose large scale applications using all of the Apache Kafka stack as well as how we view “turning the database inside out” when used in conjunction with serverless architectures.