Company
Date Published
Author
Matt Mangia, Gil Friedlis, Ram Dhakne
Word count
1255
Language
English
Hacker News points
None

Summary

Apache Kafka is a distributed commit log used as a multi-tenant data hub to connect diverse source systems and sink systems, commonly transforming ETL jobs from batch mode to near-real-time mode. It's increasingly becoming the de facto event streaming platform for enterprises across all verticals, democratizing data for both internal and external users or applications of the data. Kafka is making a huge difference in industries like transportation assets, where it can be used to track the movement of assets in real time, specifically for trams, buses, and high-speed electric trains used in the Helsinki Region Transport (HSL) system. The MQTT source produces data into a Kafka topic called vehicle-positions, which is then enriched using Kafka Streams, making it easy for Elasticsearch to consume and display on a dashboard in Kibana. This entire workflow can be accomplished with minimal effort on the development side. After enrichment, the data is written into the topic vehicle-positions-enriched, where it's pushed into Elasticsearch using a sink connector, setting up a dynamic template to recognize the geolocation data. The resulting real-time dashboard running on Kibana provides insights and visualizations of the geolocation data, making it possible for companies to be less reactive and more proactive about meeting the demands of tomorrow starting today.