Big Fish Games is a leading producer and distributor of casual and mid-core games. They have distributed more than 2.5 billion games to customers in 150 countries, representing over 450 unique mobile games and over 3,500 unique PC games.
Apache Kafka is used in our pipeline to process data generated across game play. Recently, we introduced real-time analytics of game data using Kafka Streams integrated with Elasticsearch. This allows us to monitor the results of live operations (aka, live ops such as weekend events, limited time offers and user acquisition campaigns) and to make changes to these events after they have gone live. The result is a better game experience, better control of the game’s economy and overall better optimization of the live ops.
This presentation will include a detailed explanation of how we used Kafka Streams to transform raw data into Elasticsearch documents, and how the application was scaled to over a million daily active users. It will also touch on the limitations discovered with Kafka Connect integration with Elasticsearch and how to use Elasticsearch bulk processing with Kafka Streams. The presentation will also discuss how Dropwizard can provide a framework for monitoring and alerting a Kafka Stream application. Lastly, a demonstration of the real-time dashboards which are powered by Kafka Streams will be included.