February 2020 Summaries

The design of Apache Kafka systems involves configuring parameters such as replication factor, batch size, linger.ms, compression type, acks configuration, and consumer group settings to achieve high throughput while bounding end-to-end latency. The goal is to minimize the impact of replication overhead on commit time and reduce queueing delays. By monitoring and tuning client configurations, scaling brokers, and optimizing batching, developers can ensure that their Kafka systems meet application requirements for both throughput and latency.

The text discusses the implementation of a streaming data processing system using Apache Kafka, Confluent Platform, and Kafka Streams. The system is designed to process large amounts of flight data in near real-time, providing insights into air traffic patterns, airline performance, and other relevant metrics. The architecture is based on event-driven design principles, with each component having a specific responsibility and being decoupled from others through the use of message brokers like Kafka. The system also leverages best practices such as code splitting, lazy loading, and caching to minimize its impact on end-user devices. Additionally, the text highlights the potential applications of event stream processing in various industries, including fintech, insurtech, e-commerce, and smart farming, where it can help provide real-time data-driven insights and improve business operations.

Confluent has released its most significant update yet, Confluent Platform 5.4, which includes features such as Role-Based Access Control (RBAC), Structured Audit Logs, Multi-Region Clusters, and Schema Validation to help enterprises take Apache Kafka® and event streaming into production. The company has also reached over 100 supported connectors for getting data in and out of Kafka, with most developed by Confluent and some by technology partners. These connectors are verified through a program for independent software vendors (ISVs) and partners to ensure interoperability and functionality with Kafka and the Confluent Platform. The company has also revamped its online marketplace, Confluent Hub, to make it easier for users to browse, search, and filter connectors and other plugins based on their data movement needs for Kafka.

Confluent Cloud is now available on Azure Marketplace, offering developers a fully managed service for Apache Kafka that integrates seamlessly with their existing billing account, eliminating the need to manage multiple bills. This move makes it easier for developers to get started with Confluent Cloud, which provides a serverless experience with elastic scaling and pricing based on consumption units. With Confluent Cloud, developers can focus on building event streaming applications without worrying about operational complexities, and can take advantage of tools like Confluent Schema Registry, connectors, and ksqlDB to simplify development. The service is now accessible through Azure Marketplace, allowing users to easily spin up Confluent Cloud with their existing Azure credits, eliminating the need for a separate credit card payment.

The announcement of ksqlDB 0.7.0 highlights significant improvements, including highly available state, security enhancements, and support for a broader range of data types and expressions. A key feature is the ability for queries to be rerouted to standby servers during a cluster rebalance, mitigating unavailability issues, although users must manage potential staleness in results. New functions like count_distinct and cube have been introduced to enhance data analysis capabilities, while the system now supports a wider variety of key data types beyond strings. Additionally, working with compound data has been simplified with first-class constructors for arrays, maps, and structs, allowing for more intuitive data manipulation. The update also hints at future enhancements in app development ergonomics and query layer performance for the upcoming 0.8.0 release.

Building on the first part of the series, this section delves into processing osquery logs using the Confluent Platform and ksqlDB to detect anomalous behavior with machine learning. By training a Latent Dirichlet Allocation (LDA) model on osquery logs, the system can identify deviations in behavior, categorizing them as GOOD, BAD, or UGLY based on their scores. The GOOD logs, indicating normalcy, are fed back into the model for retraining, while BAD logs, suggesting suspicious activity, and UGLY logs, requiring further analysis, can be routed to security systems for deeper investigation. The architecture uses a combination of batch and streaming pipelines, with a Lambda Architecture approach, to manage model training and real-time log scoring. This setup facilitates the integration of the Confluent Platform, Kafka Connect, and ksqlDB to create a streamlined SIEM pipeline capable of real-time alerting and investigation, laying a foundation for scalable security solutions.

Apache Kafka is often used alongside Elasticsearch to perform log exploration, metrics monitoring, and alerting, data visualisation, and analytics. ksqlDB is an event streaming database built on top of Apache Kafka that complements the Elasticsearch ecosystem while offering different approaches to handling certain scenarios. Both ksqlDB and Elasticsearch are powerful tools for data enrichment and transformation as well as alerting before ingestion into Elasticsearch where the search and analytics can take place. ksqlDB allows users to write SQL queries on streams from various sources, create derived streams and materialized tables, and perform database-like lookups on those materialized tables. It simplifies stream processing by providing a declarative language like SQL, making it an advantage over lower-level stream processing systems in the data ecosystem. ksqlDB can be used to build event-driven applications that transform, enrich, and react to data as it arrives, complementing Elasticsearch where data is sent for long-term storage and analytics. Typically, Kafka is deployed in front of Elasticsearch using Kafka Connect to push data from selected Kafka topics to Elasticsearch.

Osquery, developed by Facebook, is an open-source tool that collects audit log events from operating systems using SQL commands to simplify querying OS-level information. Its integration with modern security information and event management (SIEM) systems involves using the Confluent Platform to channel osquery data streams into Apache Kafka, with support for operating systems like Windows, macOS, CentOS, and FreeBSD. While osquery's default Kafka producer logger plugin has limitations, a more robust solution involves using a custom Python extension with librdkafka, which enables seamless log publication to Kafka, Confluent Platform, and Confluent Cloud, bypassing the need for a Kafka connector. This setup allows creating ksqlDB streams that join data from osquery's process and open file logs, providing detailed insights into system activities via Confluent Control Center. The blog outlines the initial steps of leveraging osquery with the Confluent ecosystem, with future plans to apply machine learning for anomaly detection.

A centralized database can become a bottleneck as a company grows, leading to slow queries, increased engineering complexity, and decreased innovation. A materialized cache, which precomputes query results and stores them for fast access, can help alleviate these issues. By leveraging a construct like ksqlDB, an event streaming database that supports materialized views, companies can scale out multiple caches concurrently, reducing pressure on their databases. With ksqlDB's native Kafka integration, developers can easily replicate this pattern of scaling out distributed caches. The technology allows for the creation of materialized caches that can be queried in real-time, without requiring computation, and provides a clean way to forward change capture events from databases into Kafka. By introducing core building blocks like materialized caches, companies can build powerful streaming apps with a small set of SQL statements.

Kafka Summit London, scheduled for April 27th and 28th, is an ideal opportunity for those looking to deepen their understanding of Apache Kafka, featuring 56 sessions across four non-themed tracks, allowing for a more dynamic selection of high-quality talks. The summit includes keynotes from influential figures like Jay Kreps and Sam Newman, who will discuss the future of Apache Kafka and distributed systems. Although thematic tracks have been replaced with numbered ones, session tags offer an easy way to identify relevant topics. Participants can also engage with the Confluent Community Slack for assistance with crafting talk submissions for the upcoming Kafka Summit in Austin. Attendees are encouraged to register using a promotional code for a discount, and those unable to attend in London have the opportunity to catch up on related AI and data innovations at other events, such as AWS re:Invent 2024.