Kafka Summit Logo
Organized by

Kafka Summit NYC 2019

April 2, 2019 | New York City

Use Apache Gradle to Build and Automate KSQL and Kafka Streams

Session Level: Intermediate

KSQL is an easy-to-use and easy-to-understand streaming SQL engine for Apache Kafka built on top of Kafka Streams. The ability to write streaming applications using only SQL makes Apache Kafka available to a whole range of new developers and potential use cases, either as a stand-alone solution, or as a single component to a broader Kafka Streams implementation. Inspired by a customer project now in production, experience the lifecycle of a streaming application developed using KSQL and Kafka Streams. With Apache Gradle as our build framework, we’ll explore the open-source Gradle plugin we built during this project to improve developer efficiency and automate the deployment of KSQL pipelines, user-defined functions, and Kafka Streams microservices.

We’ll demonstrate the deployment process live, and discuss design decisions around incorporating SQL-based processes into an overall streaming application.

Key Takeaways

1. KSQL is a natural choice for expressing data-driven applications, but it may not naturally fit into established DevOps processes and automations.

2. We built an open-source Gradle plugin to handle all aspects of deploying a Kafka-based streaming application: KSQL pipelines, KSQL user-defined functions, and Kafka Streams microservices.

3. KSQL pipelines can be deployed using either a server start script, or the KSQL REST API, and our Gradle plugin fully supports both options.

We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.