In this talk we will present how we in Schibsted have set up a new global streaming data platform using Kafka and Kafka Streams, replacing a homegrown solution based on Kinesis and micro batches in Amazon S3.
Schibsted is a global media and marketplaces company with over 200 million users across the world. With our new streaming platform, we aim to deliver better performance and enable new features, such as self-serve for our data consumers.
In this talk, we will present some of the ways this new platform enables collaboration across Schibsted, as well as some of the challenges we have faced. A number of collaborations with various teams in Schibsted are underway to build projects on top of this new platform. Examples include building tools for experimentation on our sites, visualization of user behavior and data quality checks. Our long-term goal is to provide a self-serve platform for real-time processing of data, enabling our data users to quickly create new data-driven applications. Data and analytics is a central part of Schibsted’s strategy, and we believe the streaming platform will play a significant role in building a global data-driven organization.