Kafka Summit Logo
Organized by

Kafka Summit San Francisco 2018

Streaming platforms at massive scale.

October 16-17, 2018 | San Francisco

How Big Fish Games Developed Real-Time Analytics Using Kafka Streams and Elasticsearch

Session Level: Intermediate
Video & Slides

Big Fish Games is a leading producer and distributor of casual and mid-core games. They have distributed more than 2.5 billion games to customers in 150 countries, representing over 450 unique mobile games and over 3,500 unique PC games.

Apache Kafka is used in our pipeline to process data generated across game play. Recently, we introduced real-time analytics of game data using Kafka Streams integrated with Elasticsearch. This allows us to monitor the results of live operations (aka, live ops such as weekend events, limited time offers and user acquisition campaigns) and to make changes to these events after they have gone live. The result is a better game experience, better control of the game’s economy and overall better optimization of the live ops.

This presentation will include a detailed explanation of how we used Kafka Streams to transform raw data into Elasticsearch documents, and how the application was scaled to over a million daily active users. It will also touch on the limitations discovered with Kafka Connect integration with Elasticsearch and how to use Elasticsearch bulk processing with Kafka Streams. The presentation will also discuss how Dropwizard can provide a framework for monitoring and alerting a Kafka Stream application. Lastly, a demonstration of the real-time dashboards which are powered by Kafka Streams will be included.


We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.