At Bloomberg, we are building a streaming platform with Apache Kafka, Kafka Streams and Spark Streaming to handle high volume, real-time processing with rapid derivative market data. In this talk, we’ll share the experience of how we utilize Kafka Streams Processor API to build pipelines that are capable of handling millions of market movements per second with ultra-low latency, as well as performing complex analytics like outlier detection, source confidence evaluation (scoring), arbitrage detection and other financial-related processing.
- Our system architecture
- Best practices of using the Processor API and State Store API
- Dynamic gap session implementation
- Historical data re-processing practice in KStreams app
- Chaining multiple KStreams apps with Spark Streaming job