In the “Talking Traffic Partnership” (https://www.talking-traffic.com/en) the Dutch Ministry of Infrastructure and Environment collaborates with several public and private parties to deliver up-to-date traffic information from a wide variety of data sources to road users via smartphones and personal or onboard navigation systems. KPN was selected as IT partner for Talking Traffic, and Klarrio was commissioned by KPN to build a platform that could: – Act as a secure streaming information exchange between the Talking Traffic partners. – Deliver personalized subsets of selected data streams to millions of connected client devices and applications in real time.
In this talk we will walk you through the production platform, describing how our partners run containerized Kafka applications against a secured multi-tenant Kafka setup to amass a wealth of traffic information.
We will show:
– How we manage tenants, data streams, and access to streams.
– How we protect the platform from rogue applications.
– How data provenance is handled in the platform.
– How tenants are given insight into the operation and performance of their Kafka applications. We then show how we created a scalable messaging layer on top of this information backbone, enabling us to disseminate relevant traffic information towards millions of connected road users over MQTT.
We focus on: – How we deliver millions of individualized subsets of the data on Kafka with minimal data amplification. – How we implement MQTT features like wildcard subscriptions and retained messages. By the end of the session you will have learned a way to deploy Kafka in large-scale, multi-tenant environments, and how to quickly and securely stream data from shared internal Kafka topics to consumers outside of the platform.