Imagine you must make data-driven decisions in real-time, whether that’s detecting anomalies and fraudulent activities in data feeds, monitoring application behavior and infrastructure, processing CDC information of your databases, or doing real-time ETL. Stream processing is the solution, but unfortunately the world of stream processing still has a very high barrier to entry. Today’s most popular stream processing technologies require the user to write code in programming languages such as Java or Scala. This hard requirement on coding skills is preventing many companies to unlock the benefits of stream processing to their full effect.
In this talk, I introduce the audience to KSQL, the open source streaming SQL engine for Apache Kafka. KSQL provides an easy and completely interactive SQL interface for data processing on Kafka — no need to write any code in a programming language. Instead, all you need is a simple SQL statement such as SELECT * FROM payments-kafka-stream WHERE fraudProbability > 0.8. KSQL brings together the worlds of streams and databases by allowing you to work with your data in a stream and in a table format. Built on top of Kafka’s Streams API, KSQL supports many powerful operations including filtering, transformations, aggregations, joins, windowing, sessionization, and much more. It is open source (Apache 2.0 licensed), distributed, scalable, fault-tolerant, and real-time. You will learn how KSQL makes it easy to get started with a wide range of stream processing use cases such as those described at the beginning. We cover how to get up and running with KSQL, explain its core data abstractions, walk through several use cases, describe its deployment options, and also explore the under-the-hood details of how it all works.