May 2018 Summaries

The Apache Kafka connector based on Xenon, a high-performance storage and analytics engine, is being showcased for streaming data from Kafka into Xenon, leveraging the Kafka Connect framework. Xenon is designed to manage large datasets efficiently with low latency and scalability, achieving optimal I/O performance and compute efficiency through just-in-time compilation and direct I/O to PCIe SSDs. The connector offers features such as multi-tenancy via a connector interface to Apache Spark, robust data formats including Avro and JSON, and high-performance capabilities demonstrated by ingesting 470GB of data at 3.5 million messages per second on a commodity server with NVMe SSDs for storage.

The event streaming service provider Confluent is launching a self-service offering of its cloud-based platform, allowing developers to build applications on a Kafka-based event streaming platform without the operational burden of running it themselves. This new service is designed to be fully managed, low-cost, and available on multiple public cloud providers, including Google Cloud Platform. The launch addresses the needs of development teams that require a self-service offering with minutes-level provisioning and usage-based charging. With Confluent Cloud, developers can focus on building applications without worrying about the operational burden of running a streaming platform, and access a broad ecosystem of tools and services to support their use cases.

In the wake of the Kafka Summit London, a wealth of session videos has been made available to the Apache Kafka community, providing insights into the evolving landscape of streaming platforms. Key among these is Neha's keynote, which outlines a five-step journey organizations undergo when adopting streaming systems, highlighting the transformative impact of streaming data on businesses. Talks by industry leaders from Audi and Apple showcased the integration of Kafka in their operations, emphasizing the shift from traditional manufacturing to data-driven strategies, while operations-focused discussions led by experts like Gwen Shapira offered practical advice for managing Kafka systems effectively. The Summit underscored the shift from static data ingestion to dynamic, real-time data architectures, positioning Kafka as a crucial tool for modern enterprises. As the community looks forward to future gatherings, such as Current 2024 in Austin, the emphasis remains on collaboration and knowledge sharing to drive the platform's evolution.

With the release of Confluent Platform 7.7, which includes the production-ready KSQL streaming SQL engine for Apache Kafka, users can more easily manage real-time data streams, though new users might still require guidance. The platform offers an intuitive interface and various tools like Confluent Control Center for monitoring KSQL queries, appealing to both novice and experienced users. Upcoming resources, such as the "Level Up Your KSQL" video series, aim to deepen understanding of essential KSQL concepts, including streams and tables, data aggregation, and scalability. Additionally, the new platform features enhanced security with OAuth support and introduces Confluent Platform for Apache Flink, alongside a new connector, while also exploring integration with Amazon OpenSearch through OpenSearch Ingestion.

The Confluent Operator is a Kubernetes Operator that enables users to deploy and manage Apache Kafka on Kubernetes, overcoming the challenges of stateful applications such as maintaining stable identities for nodes, retaining node states across failures, and restoring application lifecycles. It leverages Custom Resource Definitions and Operators to provide deployment and management automation for Kafka and the Confluent Platform on Kubernetes, addressing nuances specific to Kafka, such as configuration management, rolling cluster restarts, and data balancing. The Operator will support popular Kubernetes distributions and will be released in conjunction with a reference architecture, deployment templates, and Confluent Platform docker images, aiming to make streaming data ubiquitous by combining the flexibility of Kubernetes with the scalability of Apache Kafka.

Confluent has introduced a preview release of its platform, which includes powerful new capabilities for KSQL and Confluent Control Center, including a brand new KSQL UI that provides a graphical interface to view and edit queries, stream data, and tables. The control center also offers improved features such as viewing broker configurations, inspecting Kafka topics, monitoring consumer lag, and enabling non-windowed usage of aggregation functions on tables. Additionally, the preview release includes improvements to the KSQL REST API, including support for specifying timestamp formats when creating streams or tables, and preventing dropping of active streams or tables without terminating dependent queries. Users can share their feedback through the Confluent Community Slack channel or by contributing to the KSQL project on GitHub.