Kafka Summit Logo
Organized by

Kafka Summit London 2019

Streaming platforms at massive scale.

May 13-14, 2019 | London

Industry-ready NLP Service Framework Based on Kafka

Session Level: Intermediate

Natural Language Processing (NLP) focuses on the analysis and understanding of textual information either in written or spoken form. In recent years, NLP technologies has become business critical due to the overwhelming and ever-increasing amount of textual information to get customer or business process insights, but also to empower novel user experience by creating dialogue-based digital assistants understanding customer language. In our talk, we will explore how NLP use cases significantly benefit from stream processing and event driven architectures. We will present the NLP Service Framework representing a stream processing framework using Kafka in which NLP tasks run as microservices orchestrated in pipelines to perform complex end-to-end services. In the NLP Service Framework, Kafka is being used to orchestrate data flows containing of all kinds of textual information in different topics related to specific use cases. Different Kafka Streams based processors subsequently call NLP services to analyze and annotate the textural information within the data flows. Various applications like search-based application based upon Elasticsearch and Kibana or analytical databases eventually consumes the textual information that is augmented with annotations and inferred results of the NLP Services. Two important requirements of the NLP Service Framework are efficient communication between different services using REST interfaces and interoperability among services implemented in different languages such as Java or Python. We implement the gRPC framework and use ProtoBuff as data format to ensure both requirements. This Kafka-based architecture enables us to specify domain-specific but isolated end-to-end NLP services and guarantees highly scalable and robust handling of high volume of textual data from different BMW domains along the value chain, including customer, process, and vehicle data.


We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.