May 2020 Summaries

Confluent is introducing features that enable Kafka clusters to scale more easily, allowing businesses to meet real-time customer expectations without the need for detailed capacity planning. This scalability is expected to be beneficial for both self-managed and cloud-native environments. The new features include Operator, which automates the process of adding brokers, Tiered Storage, which decouples compute from storage resources, and Self-Balancing Clusters, which optimizes resource utilization. Together, these features simplify the scaling process, making it faster and more efficient, and enabling businesses to take advantage of cloud-native data systems that can scale up as demand peaks and down as demand falls.

The text discusses the significance of securing Apache Kafka and Confluent Platform, which are essential for data streaming across enterprises, by focusing on authentication, authorization, encryption, and monitoring. Authentication methods like SASL and mTLS are crucial for verifying users and applications, while authorization via ACLs and RBAC determines permitted actions. The text emphasizes the importance of encrypting data in transit to prevent unauthorized access and introduces Confluent's Secret Protection to safeguard sensitive configuration data. Additionally, it highlights the role of structured audit logs and Confluent Control Center in monitoring and alerting on security events, as well as the necessity of data governance through schema validation and quota management. The text also touches on compliance with industry standards and announces new features in Confluent Platform 7.9, including an Oracle XStream CDC Connector and client-side field-level encryption.

The blog post explores the potential of using Apache Kafka and its ecosystem to process Wi-Fi packet capture (pcap) data collected via a Raspberry Pi, which streams the data to Kafka. The author sets up a pipeline that processes, joins, aggregates, and streams the pcap data to various datastores like Neo4j and PostgreSQL. By leveraging tools such as ksqlDB and Kibana, the author visualizes and analyzes the data to uncover patterns and insights, such as device interactions and network probing behavior. The post also highlights the integration of additional data sources, like a MongoDB-stored device lookup, to enrich the data streams and enable more detailed analysis. Finally, it demonstrates the flexibility and power of Kafka's platform to manage high volumes of data and facilitate complex data processing tasks, including graph analysis with Neo4j and aggregation with ksqlDB, providing a comprehensive framework for managing streaming data.

The blog post explores the development of an event-driven application using ksqlDB, Kafka Connect, and Elasticsearch, demonstrated through a clickstream demo. It illustrates how businesses can analyze user behavior and track activities on their websites by collecting and processing streaming data. The demo showcases the use of SQL statements to create streams and tables from clickstream data, enabling real-time analysis of user interactions and errors. By embedding Kafka Connect functionality, the application can export processed data to Elasticsearch for visualization using Grafana, without extensive coding or infrastructure management. The post emphasizes the ease of setting up test data scenarios with the Kafka Connect Datagen Connector and highlights the potential of using these technologies to scale AI-powered solutions within organizations.

Confluent Cloud, a fully managed event streaming platform, now supports Protobuf and JSON Schemas in addition to Apache Avro, enhancing its Schema Registry capabilities. This update allows developers to manage schemas across Kafka topics, ensuring easy integration and development across on-premises and cloud deployments. The blog post provides a comprehensive guide on implementing Protobuf with Confluent Cloud, covering steps to enable Schema Registry, generate client configurations, and create producer and consumer applications that handle sensor data. It also explores tools like Maven plugins for schema management and ksqlDB for streaming applications, highlighting the platform's ability to streamline schema governance and enhance application development efficiency. Additionally, Confluent Cloud's CLI tools facilitate integration with CI/CD platforms, offering robust solutions for managing events and schemas.

Kafka Streams, a component of Apache Kafka, has been utilized by Walmart to build a high-availability, low-latency customer data platform for applications such as fraud detection, which requires rapid data processing and stringent service level agreements. The platform faces challenges common in distributed systems, such as balancing consistency and availability, particularly during cloud environment disruptions and system patches. Enhancements like KIP-535 and KIP-562 have been implemented to allow state stores to serve data from standby servers during rebalancing, reducing downtime and improving load balancing. Additionally, KAFKA-9169 addresses bugs related to unnecessary restoration processes, further increasing system availability. These improvements enable Kafka Streams to move closer to replacing traditional databases by providing high availability and low latency, reducing the need for multiple event streaming engines and databases in application development. The advancements in Kafka Streams, including interactive queries and versioned key-value state stores, are part of its evolution to challenge traditional databases and streamline developer workflows.

KIP-500` is a proposal to move metadata management into Apache Kafka itself, breaking its dependency on external systems like ZooKeeper. This change aims to reduce complexity and improve efficiency by storing metadata in a partition inside Kafka, eliminating the need for an external metadata system. The project introduces a Raft protocol for the metadata quorum, allowing for self-managed replication and minimizing controller failover time. KIP-500 mode will be experimental initially but is expected to become the default as it gains confidence. The proposal also includes bridge releases to enable zero-downtime upgrades from older ZooKeeper-based versions of Kafka. By simplifying architecture and leveraging the event log abstraction, KIP-500 aims to prove that Kafka needs no external keeper like ZooKeeper.

The release of ksqlDB 0.9.0 introduces significant enhancements, including the ability to perform multiple joins within a single statement, improved LIKE expressions, and various usability improvements. This update allows more efficient execution of join operations, reducing the need for intermediate streams and simplifying the process of generating reports from multiple data sources, such as customers and orders. With the new LIKE expression capabilities, users can now match patterns more flexibly, using the % and _ characters, and specify an escape character. The release also includes a new COALESCE function and various bug fixes, all detailed in the changelog. Users are encouraged to join the Confluent Community Slack channel for further engagement and to leverage these improvements in scaling AI-powered solutions through real-time data processing and event-driven architecture.

The text explores the use of Apache Kafka and ksqlDB to process and query streaming data without relying on external datastores. It describes a system that captures Wi-Fi packets and processes them to provide real-time information about connected devices via a Telegram bot, showcasing ksqlDB's ability to create stateful aggregates and perform pull and push queries. The approach highlights Kafka's role as a distributed, scalable, and fault-tolerant platform that can serve as both a message broker and a data store, further enhanced by ksqlDB's SQL-based stream processing capabilities. The article also delves into integrating MongoDB with Kafka to enrich data streams and emphasizes the flexibility of event-driven architectures in building systems that respond to real-time data, demonstrating how ksqlDB's materialized views and integration with other data sources can simplify complex query operations and data management.

The Apache Kafka rebalancing protocol is designed to ensure that consumer groups can rebalance their partitions without causing downtime or significant performance degradation. The current eager rebalancing protocol, which was introduced in version 2.3 of Kafka Streams, stops the world and forces all consumers to revoke their partitions before rejoining the group. However, this approach has drawbacks, including downtime for all partitions and all members of the group, as well as potential issues with partition ownership and migration. To address these challenges, a new incremental cooperative rebalancing protocol was introduced in version 2.4 of Kafka, which allows consumers to continue processing records while reassigning their partitions incrementally. The new protocol is designed to be more efficient and less painful than the eager protocol, while still maintaining at-least-once and exactly-once processing guarantees. By using a cooperative partition assignor, such as the CooperativeStickyAssignor, consumers can ensure that their partitions are reassigned safely and efficiently, without causing downtime or performance degradation. The new protocol has been extensively tested in Kafka Streams and ksqlDB applications, and has shown promising results in reducing rebalance times and improving overall availability.

Confluent has introduced a new agent-based integration with Datadog for monitoring its Confluent Platform, allowing mutual customers to monitor Confluent alongside their entire IT infrastructure. While the Confluent Control Center remains a robust option for a comprehensive view of the platform, the partnership with Datadog facilitates easier integration across the IT stack. The integration, available from Agent versions v6.19 and v7.19, simplifies the monitoring of Confluent Platform services by pulling in JMX metrics and host-level system metrics. Users can customize the default dashboard provided by the integration to suit their business needs. The blog also highlights the complementary value of using Datadog with Control Center for resolving issues like failed tasks and anomalies in ksqlDB queries. Additionally, Confluent Platform 7.9 introduces new features such as the Oracle XStream CDC Connector and Client-Side Field Level Encryption, enhancing its functionality.

Confluent is working on Project Metamorphosis, an initiative that aims to bring together two major shifts in the world of data: event streaming and cloud computing. Event streams are a key aspect of Confluent's business, but they often involve manual configuration and management. The project seeks to simplify this process by providing a seamless experience for users. Confluent Cloud is being updated with features such as elasticity, self-service expansion, automatic data balancing, and autoscaling, which will allow users to scale their clusters up or down as needed without manual intervention. These changes are expected to make cloud-native data systems more accessible and user-friendly. The project also aims to bring cloud-like elasticity to private cloud and on-prem environments through the Confluent Operator and contributions to Apache Kafka itself.

ksqlDB has improved its high availability by implementing heartbeating for failure detection, allowing pull queries to access underlying state during rebalancing while Streams standby replication brings the new server up to speed with the failed server, and enabling lag-aware routing where servers can quickly determine which other servers are alive and how caught up their tables are. This enables highly available pull queries by specifying a maximum allowed offset lag for each query, allowing users to trade off some consistency for more availability. The new design also allows ksqlDB to be configured as an AP system, trading inconsistency within specified bounds for availability, though once the inconsistency exceeds the bounds, it returns to being a CP system.

The text discusses how Walmart has innovatively tackled the challenges of modern retail inventory management by implementing a real-time inventory system using Apache Kafka, which is crucial for adapting to evolving consumer shopping patterns that span multiple channels. To address the complexities of handling diverse event sources and data types, Walmart developed a canonical data approach and a smart transformation engine to create a unified inventory view, allowing seamless integration and acceleration of data delivery. The text emphasizes the importance of scalability in event streaming architectures, highlighting strategies for optimizing partitions, producer, and consumer configurations, while also ensuring database design effectively handles the data flow. Additionally, it underscores the critical need to align Kafka with database partitioning strategies for optimal performance, particularly with Cassandra, and reflects on the successful application of Kafka at Walmart, alongside a separate example of Skai's use of Kafka and Confluent Cloud to enhance their ad-campaign dashboard. The narrative concludes by noting the importance of understanding and capturing meaningful events to drive successful outcomes at scale, with a disclaimer that the views expressed are those of the author and not necessarily of Walmart Inc.

The text discusses the continuous improvements made by the ksqlDB Engineering Team to enhance ksqlDB for production availability in Confluent Cloud, focusing on maintaining query compatibility across different versions. It explains the challenges of upgrading ksqlDB while ensuring persistent queries remain functional, highlighting the issues with previous solutions that relied on compatibility-breaking configurations. The team has introduced execution plans that decouple the construction of Kafka Streams topologies from the ksqlDB engine, enabling changes without affecting running queries. This approach, which involves JSON-serializable execution plans, allows for easier management of compatibility and facilitates future developments without breaking existing functionalities. The blog post also touches on the potential for evolving ksqlDB's grammar and semantics and the importance of maintaining external compatibility for users. The article concludes by inviting readers to explore further improvements and engage with the community, emphasizing the importance of real-time data processing in AI-powered solutions.