June 2020 Summaries

The text discusses how to handle "poison pill" records in Apache Kafka applications, which are records that always fail to be deserialized by the consumer. The author shares lessons learned from their experience at ING and provides guidance on how to configure Spring Kafka to survive such scenarios. They explain that the problem occurs when the producer serializer and consumer deserializer are incompatible, leading to a deserialization exception. To handle this, they recommend using the `ErrorHandlingDeserializer`, which delegates to real deserializers if the delegate fails to deserialize the record content. The author also discusses how to configure the `DeadLetterPublishingRecoverer` to publish the value of the poison pill to a dead letter topic, allowing for inspection of the data. They emphasize the importance of proper monitoring and logging to prevent the consumer application from "eating" server disk space or other services becoming unhealthy due to a full disk.

The text discusses the cost-effectiveness of the Confluent Platform in managing Apache Kafka, highlighting how its features significantly reduce the total cost of ownership for businesses. Confluent Platform offers essential features like Tiered Storage and Self-Balancing Clusters, which optimize infrastructure use and cut storage costs by offloading older data to cheaper object storage and automating data distribution, respectively. Multi-Region Clusters enhance disaster recovery with faster recovery times and reduced downtime costs. Operational costs are minimized through pre-built connectors, a GUI for management, and the Confluent Operator, which automates tasks such as deployment and scaling on Kubernetes. Additionally, Confluent's Professional Services and Support further reduce the time and effort required for platform development and maintenance, offering 24/7 assistance to avoid and resolve issues efficiently. Overall, these features and services can lower the total cost of ownership by up to 40%, potentially covering the cost of a Confluent subscription and improving the effectiveness of Kafka initiatives.

Confluent Cloud has announced the general availability of its fully managed Snowflake sink connector, which facilitates seamless data integration between Apache Kafka and Snowflake across major cloud platforms like AWS, Azure, and Google Cloud. This development aims to reduce operational overhead by eliminating the need to manage a Kafka Connect cluster. Snowflake's Cloud Data Platform provides a robust infrastructure for cloud-based data warehousing and diverse data workloads, allowing organizations to derive insights from their data efficiently. To utilize the Snowflake sink connector, users must have a Confluent Cloud account and a Snowflake account located in the same cloud region to avoid additional data movement charges. The blog post outlines a scenario involving a food delivery service where data from user interactions on a new profile page is analyzed using the Snowflake sink connector. By creating pipelines with minimal operational burden, users can benefit from Confluent Cloud's managed event streaming services, enabling the retrieval and integration of data without the complexities of infrastructure management.

ksqlDB 0.10.0 introduces several significant updates, including a first-class Java client and enhanced key support, enhancing its integration with Apache Kafka. The Java client, part of a series of new clients, enables more efficient interaction with ksqlDB through support for push and pull queries, as well as data insertion into existing streams. This version removes previous limitations on Kafka key namespaces and allows for any key name, facilitating a more flexible handling of data. The update also introduces new built-in functions for stream processing, such as string manipulation and array handling functions, while dropping the "WITH KEY" syntax in favor of more straightforward key naming. These enhancements aim to simplify and improve the usability of ksqlDB for stream processing applications, with more details and updates available in the changelog and community resources.

The text discusses the increasing demand for immediacy in consumer experiences and how software, particularly event stream processing, is pivotal in meeting these expectations. Companies are becoming software-driven, utilizing stream processing to automate processes and provide real-time feedback, significantly enhancing customer interaction. However, the complexity and cost of developing and maintaining these systems can be prohibitive, often requiring multiple distributed systems and highly skilled engineers. The introduction of ksqlDB, an event streaming database developed by Confluent, aims to simplify this process by integrating event capture, processing, and querying into a single system, reducing the cognitive and financial burden on companies. ksqlDB supports the creation of data pipelines and materialized views, facilitating the development of event-driven applications and potentially making sophisticated stream processing accessible to more businesses. The text concludes by highlighting the potential of ksqlDB to help companies embrace software solutions more efficiently, promising a cost-effective approach to event streaming that could transform how businesses operate.

Company mergers often require the integration of disparate data pipelines, and event streaming plays a crucial role in this process. Digital companies, such as ride-sharing and gaming firms, capture real-time business events, and when they merge, they face challenges in unifying their data streams to accommodate increased scale and volume. Confluent Cloud, built on Apache Kafka, offers a solution by providing a fully managed event streaming service with a broad ecosystem of connectors, facilitating the integration across platforms with minimal downtime. By centralizing data streams via Confluent Cloud, companies can continue to make real-time operational decisions and merge infrastructure efficiently. Examples include using Kafka Connect to integrate real-time data feeds into a centralized Kafka cluster and employing ksqlDB to compute business metrics post-merger. The blog illustrates this process by detailing the integration of an Amazon Kinesis data stream into Confluent Cloud using a fully managed Kinesis source connector.

A managed service like Confluent Cloud can offer several benefits over DIY, including reduced upfront costs, scalability without worrying about capacity planning, routine maintenance, and troubleshooting. However, there are also potential drawbacks such as vendor lock-in, limited control over the underlying infrastructure, and the need to consider usage-based billing and network costs. To make an informed decision, it is essential to carefully calculate the total cost of ownership (TCO) for both DIY and managed services, including engineering time, capacity planning, routine maintenance, and potential downtime costs. Additionally, considering factors such as performance tuning, vendor support, and scalability can also impact the overall cost-effectiveness of a chosen solution.

Organizations often use monitoring tools like AppDynamics, Datadog, and Prometheus to oversee the performance of critical applications, integrating multiple data sources for a consolidated view. This tutorial outlines creating a pipeline that utilizes Kafka Connect to stream Confluent Cloud metrics into AppDynamics, leveraging Confluent's managed Apache Kafka service. Kafka Connect functions through source and sink connectors, with a provided example of a Confluent Cloud Metrics Source project that reads from the Metrics API and streams data into Kafka, ultimately forwarding it to various monitoring solutions. AppDynamics is highlighted for its ability to baseline metrics and alert on abnormal behavior. Installation and configuration steps for both the source connector and AppDynamics sink connector are detailed, including schema management and message transformation processes, culminating in a system that streams metrics into AppDynamics, allowing for application health monitoring and dashboard creation.

Confluent is launching its Project Metamorphosis initiative, aiming to bring simplicity of cloud systems to event streaming, with a focus on cost-effectiveness and ease of use, targeting both small-scale and large-scale users of Kafka. The company's Confluent Cloud platform offers three types of clusters optimized for different use cases, including Basic, which allows users to pay only for the amount of Kafka used, making it a lightweight option with low costs for even the smallest use cases. Additionally, Confluent Cloud provides managed connectors, ksqlDB, and schema management using Confluent Schema Registry, among other features, to help reduce hardware needs, drive better utilization, and support multi-tenant operations. The platform also offers a usage-based model that allows users to dynamically scale up without interrupting running applications, providing elasticity and cost savings of up to 60% compared to self-managed approaches.

The COVID-19 pandemic has highlighted the challenges faced by state and federal agencies in scaling their systems to meet increasing demand, particularly with aging code bases and data stored on mainframes. Efforts to address this issue have led to the adoption of event streaming technologies such as Apache Kafka, which can free data from its mainframe quarantine and enable modern, scalable applications. Many organizations lack the resources and personnel to deploy and manage Kafka clusters on their own, but having a fully managed cloud-native service like Confluent Cloud can provide a rapid deployment option with reduced maintenance overhead. The use of role-based access control and other features such as ksqlDB enables organizations to securely manage and monitor data streams, and provides opportunities for innovative uses of the data in real-time. Several case studies demonstrate successful implementations of Kafka and Confluent Platform, including Alight Solutions, RBC, and BT Group, which have reduced costs and improved system availability.

Apache Kafka is the de facto standard for event streaming, allowing scale at a level previously unobtainable due to low cost of data streaming. However, expanding without downtime in distributed systems is challenging and requires hard work. Confluent Cloud automates infrastructure management, leveraging cloud provider services such as AWS, Azure, or GCP. The service uses the concept of CKUs (Confluent Units), which encapsulate virtual machines, disks, storage, and other resources into a single package. Clusters can be composed of 1 to N CKUs, with Confluent Cloud placing partitions intelligently in an attempt to minimize downtime during scaling operations. When scaling, Confluent Cloud automates the rebalancing process, moving partitions between brokers while ensuring minimal disruption. This allows for elastic scaling without downtime, making it a seamless experience for event streaming software as a service (SaaS).

Confluent's hack days foster collaboration across diverse teams, enabling the creation of innovative projects that address various challenges and needs within the company. These events encourage risk-taking and bottom-up inspiration, allowing employees to tackle ideas not typically included in regular sprints. Recent hack day projects at Confluent included initiatives such as enabling multilingual queries in Loopio, developing cryptographic stream integrity verification for Kafka, improving data compression using shared dictionaries, automating the cleanup of unused cloud resources with Project Roomba, and creating Conflack, a tool to preserve Slack messages in Confluence. These projects highlight the company's commitment to exploring new technologies and solutions while promoting teamwork and creative problem-solving.

The article delves into the intricacies of configuring Apache Kafka connections, emphasizing the importance of correctly setting up the advertised.listeners configuration for seamless client-broker communication. It highlights common pitfalls where initial connections succeed, but subsequent connections fail due to incorrect broker metadata, often caused by misconfigured advertised listener addresses. The text provides practical solutions for various deployment scenarios, including running Kafka and clients on different machines, within Docker containers, and in Docker Compose setups. It suggests that using Confluent Cloud can simplify the process by eliminating the need for manual configuration of advertised.listeners, thus avoiding common connection issues. The article also mentions advanced Kafka topics like Confluent Tableflow for zero ETL integration and KIP-848 for improved consumer rebalance protocols, appealing to developers seeking to enhance their Kafka implementations.

Fleets in various industries generate vast amounts of real-time telemetry data, which can be used to build fleet management solutions that simplify operations and improve customer experience. To process this data, a combination of Confluent Cloud, fully managed ksqlDB, Kafka Connect with MongoDB connectors, and MongoDB Atlas is used. The solution utilizes Voluble to mock telemetry data and integrates it with MongoDB for historical analysis. Stream processing complements various use cases in fleet management, including hazard detection, route optimization, and real-time alerting. A ksqlDB application processes events in real-time, detecting hazardous driving patterns and generating notifications. The system also captures real-time location data and harsh-braking events from the fleets, exporting them to MongoDB for further analysis and visualization. Confluent Cloud and MongoDB work well together in designing an overall fleet management solution, especially as autonomous fleets and advanced delivery systems become more prevalent.

The rise of IoT devices means that collecting, processing, and analyzing vast amounts of data has become increasingly important. Building an infrastructure to handle this data can be challenging due to factors such as variable connectivity, bursty data, and long tails of firmware versions. Apache Kafka, Flink, and MongoDB are key technologies in handling these challenges. To build a system that can scale horizontally without becoming harder to run or adding significant operational overhead, the core piece of technology is Apache Kafka, which provides resilient storage, native stream processing capabilities, and blazing-fast performance while maintaining high throughput. The system also needs to handle large messages, which can be done through parallelization, buffering, or diverting them to a "slow lane" topic. Additionally, metadata streams can provide powerful queries that help understand the state of the fleet, such as determining the relative coverage for every device in the fleet. Ultimately, managing these dataflows should not be the goal of teams building out these tools and pipeline components, but rather empowering end users to build and manage pipelines.

This post delves into the enhancements made to ksqlDB's interactive deployment mode in Confluent Cloud, focusing on maintaining the integrity of ksqlDB's internal state. In this mode, ksqlDB servers within a cluster share a common command topic, a Kafka topic that logs all data definition language (DDL) statements, ensuring every server executes the same commands and remains synchronized. The previous protocol faced issues with statement validation errors, often skipping statements which could lead to inconsistent metastore states. The new protocol treats the command topic as a log, utilizing Kafka transactions to ensure statements are never skipped and only committed transactions are processed. This guarantees consistent recreation of the metastore across server restarts and helps prevent conflicting statements by ensuring a single writer to the command topic at any given time. The integration of Kafka's transaction mechanism, using the ksql.service.id as the transactional.id, coordinates between servers and enhances reliability and scalability in cloud deployments. The post highlights the ongoing improvements and invites community contributions, indicating a commitment to evolving ksqlDB's capabilities in cloud environments.

Confluent has reached a significant milestone by declaring its command line interfaces (CLIs), the ccloud and confluent CLIs, as generally available, enhancing user experience for managing Confluent deployments in the cloud or on-premises. Both CLIs, built on a similar codebase, offer a user-friendly experience and provide extensive features for developers, operators, and architects, including full CRUD operations for Kafka clusters, Schema Registry management, and ksqlDB administration. The CLIs introduce new features like machine-readable output and long-lived authentication, allowing for more scriptable and automated use in production environments. These tools are compatible with major operating systems and support both interactive and automated use cases, reinforcing their robustness and reliability. Confluent is committed to further developments, including enhancements for data integration and permissions management, and actively seeks user feedback to improve functionality.