October 2018 Summaries

The text provides an in-depth exploration of using KSQL, a continuous query language built on Kafka Streams, to detect fraudulent transactions in real-time by leveraging stream processing. It highlights the challenges faced in fraud detection across various industries, including banking and insurance, and explains how traditional methods often required advanced programming skills. The text introduces KSQL as an accessible solution that allows users familiar with SQL to build stream processing applications, enabling horizontal scalability and resilience. Through a detailed example involving ATM transactions, the text demonstrates how to use stream-stream joins and time windows to identify suspicious transactions. It describes the process of implementing a fraud detection system using Elasticsearch for data storage and Kibana for visualization, as well as integrating customer data from a MySQL database using Kafka Connect and Debezium to enrich the event stream with account details. The text concludes by emphasizing the scalability and flexibility of using KSQL and Kafka in developing real-time fraud detection systems and the potential for further action based on suspected fraud.

The third annual Kafka Summit San Francisco recently took place, featuring keynotes and sessions that are now available online. With over 8,000 views of the keynotes, the event highlighted topics such as building GenAI apps faster, Apache Kafka® Community growth, and cloud services for government agencies. The next Kafka Summits will be held in New York City on April 2, 2019, and London on May 13-14, 2019. Confluent Platform, the leading distribution of Apache Kafka, is available for download. Additionally, Current 2024 will bring together the data streaming community in Austin, Texas from September 17-18 to discuss topics related to Apache Kafka® and Apache Flink®.

Running Apache Kafka on Kubernetes can be beneficial in certain situations for large traditional enterprise companies, despite the complexity it introduces. It becomes the organizational path of least resistance when most applications are already running on Kubernetes, and deploying Kafka outside of Kubernetes can cause significant organizational headaches. However, successful deployment requires cooperation from skilled storage and network teams to manage shared storage devices and handle message routing to specific brokers. It is recommended against choosing Kafka as the first service to run on Kubernetes, but rather as part of a broader application ecosystem where all applications are managed with Kubernetes. With tools like Helm Charts and Confluent Operator, running Kafka on Kubernetes can be easier than other methods and offers consistent low latency when storage and network teams understand the requirements.

To determine which plane is disturbing a cat's sleep by utilizing Apache Kafka and KSQL, a streaming SQL engine for Apache Kafka, along with a Raspberry Pi and RTL2832U, a software-defined radio (SDR) device can be used. Raw ADS-B signals are processed on the Raspberry Pi, separated into location update messages or identification messages, and then relayed to Kafka topics using a Confluent REST Proxy. KSQL is used to combine the incoming stream of location data against reference data topics, creating enriched topics such as `location_and_details_stream` and `ident_callsign_stream`. These topics can be used to create dashboards displaying aircraft location on a map, manufacturer information, altitude, and destination routes, allowing for real-time analysis and monitoring.

The future of data will be self-service, with users opening consoles to define the pieces they need, format them, and spin up new endpoints such as databases, caches, microservices, or serverless functions. Event-driven architectures are becoming increasingly popular, allowing companies to store, join, aggregate, and reform data from across a company before pushing it wherever needed. There are four broad patterns of event-driven architectures: global event streaming platforms, central event stores, event-first applications, and automated data provisioning. Companies often adopt one pattern at a time, with many implementing some version of these futures already in production. Streaming platforms can cache events for defined periods or store them indefinitely, creating organizational ledgers or event stores. Event storage is required for stateful stream processing, enriching orders with customer or account information, and providing denormalized input for databases. Traditional applications often work by importing datasets into their database, but event-driven architectures push real-time events directly into microservices or serverless functions, simplifying the development process while making systems more responsive and efficient.

KSQL`, a query service developed by Confluent, is used to analyze data streams from Apache Kafka topics. It provides a powerful interface for building and exploring KSQL objects, including streams, tables, and SQL queries. The service offers various features such as metrics, error tracking, and debugging tools to help users troubleshoot issues with their queries. Additionally, it integrates with Confluent Control Center, which provides monitoring, management, and alerting capabilities for Apache Kafka environments. The service also exposes a wealth of metrics through JMX, allowing users to analyze data streams in more detail. Furthermore, it supports versioned key-value state stores, enhancing stateful processing capabilities in Kafka Streams. By using KSQL, users can gain insights into their data streams and optimize their applications for better performance and reliability.

In recent years, stream processing has become increasingly dominant in the technology industry, particularly among large internet companies like Netflix and LinkedIn. This type of software is designed to process data as it moves, allowing for real-time reactions to massive streams of event data. In traditional markets like finance, where data volumes are lower, the uptake of stream processing demonstrates a subtle shift towards reshaping IT functions. Financial institutions like Monzo, ING, Royal Bank of Canada, and Nordea have adopted stream processing technologies to adapt to changing customer needs and improve their operational efficiency. By repurposing streaming technologies used by technology giants, these firms can manipulate event streams as they move through the organization, unlocking new insights into business operations. Streaming systems also enable unified data enablement, making data a self-service commodity available on a global scale. This approach allows companies to respond instantly to customer needs and adapt to changing market conditions, ultimately leading to increased agility and competitiveness in the industry.