April 2021 Summaries

The text discusses the critical role of Apache Kafka in maintaining the availability of applications that rely on it for data ingestion and processing, emphasizing the challenges posed by potential broker failures and cluster outages. It highlights the importance of operational best practices, such as appropriate broker placement and replication settings, to mitigate these risks and ensure resilience. The text also explores various strategies for handling outages, including buffering, message storage, and load shedding, while cautioning against the pitfalls of backpressure and the complexities of maintaining order and data integrity during recovery. These strategies are particularly relevant for high-throughput applications where the cost of outages can be significant, both in terms of business loss and regulatory penalties. Additionally, the text underscores the necessity of monitoring Kafka clusters to detect and address issues proactively and suggests that organizations should consider the specific transaction models and data value when designing reliability mechanisms.

ksqlDB 0.17 marks a significant release with the introduction of several new features aimed at enhancing database schema management and query capabilities. The update includes a migration tool that allows for version control of ksqlDB schemas, facilitating integration with CI/CD pipelines by enabling automatic application of new migration files upon changes in version control systems like Git. The release also introduces a first-class timestamp data type, accompanied by functions for manipulating time data, and the ability to create inline lambda functions to transform, filter, and reduce data using SQL. Enhanced pull query support now allows for more flexible operations, such as table scans, expanding beyond key equality comparisons to include a variety of operations in the WHERE clause. These improvements enable more sophisticated querying on derived tables, although support for pull queries on tables from table-table joins is still pending. Users running ksqlDB on Confluent will automatically receive the upgrade to version 0.17, and those new to the platform are encouraged to explore ksqlDB through Confluent or its standalone distribution.

Monitoring your Apache Kafka client applications is crucial due to the potential for misbehaving clients consuming unnecessary resources of a Kafka cluster or causing scaling issues. Fortunately, Confluent Cloud fully manages the server-side of Kafka alongside other features, freeing up time and resources for managing client applications. However, tracking resource usage is essential as it sets limits on cloud services that vary by cluster type, which can lead to denied requests or delayed processing if exceeded. To monitor Kafka client applications running in the cloud, you can use various metrics exporters such as kafka-lag-exporter and ccloud-exporter, which collect data about consumer groups and present it in a scrapable format. Prometheus and Grafana are used for visualization software, while Confluent Metrics API provides actionable metrics about your Confluent deployment. The Observability for Apache Kafka Clients to Confluent Cloud tutorial showcases various scenarios, including failure scenarios, hitting usage limits, and proactively measuring request rates, with sample thresholds and dashboards provided. By monitoring client application performance, you can identify potential issues before they impact your business use cases.

Apache Kafka 2.8.0 introduces significant enhancements, including an early access version of KIP-500, which eliminates the need for Apache ZooKeeper by utilizing an internal Raft implementation, allowing for more partitions per cluster and enhanced security. The release also presents KIP-700, which decouples the AdminClient from the Metadata API, facilitating the addition of new admin features without affecting producers and consumers. Additionally, KIP-684 enables mutual TLS on SASL_SSL listeners, enhancing security, while KIP-673 restructures debug logs into a JSON format for better parsing and integration with logging tools. Kafka Streams sees improvements such as KIP-663, which allows dynamic adjustment of processing threads to optimize resource utilization, and KIP-671, which introduces a new StreamsUncaughtExceptionHandler to improve resilience by providing mechanisms to replace dead threads or shut down the system. KIP-516 introduces unique topic IDs, ensuring efficient representation and management, while KIP-572 fills resilience gaps with a retry loop for network disruptions. The release is a culmination of efforts from the community, offering numerous fixes and enhancements across the Kafka ecosystem.

The key points of the text revolve around how event streaming can be used to build active workflows in real-time, enabling business actions based on sensor data from vehicles, for instance. A concrete example is presented where a vehicle fleet sends constant sensor data, which needs to be transformed into actionable insights by a stateful connector that starts a new process instance only once for every new insight, while also informing the existing one if the problem no longer exists. This application involves challenges such as idempotency, scalability, and semantic aggregation of multiple insights, which were addressed using techniques like de-duplication, time windowing, and persistence. The solution was implemented using Apache Kafka and Camunda workflow engine, with a custom connector providing the necessary functionality to connect the streams to the workflow engine while keeping track of actions for semantic aggregation.

Confluent and Elastic are partnering to make it easier to integrate Apache Kafka and Elasticsearch, eliminating the need to manage Kafka Connect clusters and reducing operational burdens. This partnership enables organizations to deliver optimized search, real-time analytics, and data-driven applications, and has already helped customers protect their environments from evolving threats in real time. The partnership is expanding Confluent's capabilities to tackle use cases such as log analysis, full-text search, and preemptive threat mitigation, and is a key part of Confluent's strategy to help companies put their data in motion.

The text discusses two models for evaluating technology costs - Return on Investment (ROI) and Total Cost of Ownership (TCO). While ROI focuses on financial return, TCO factors in all deployment and operation costs. However, neither model provides a complete picture alone. Confluent combines both ROI and TCO with an additional factor: speed to market. Deploying Apache Kafka presents challenges that can be mitigated by using fully managed services like Confluent Cloud, which allows organizations to shift resources to higher-value tasks. The text also discusses the importance of risk costs and how a Monte Carlo simulation can model a range of outcomes for ROI calculations. By combining speed to market, TCO, and ROI, one gets a complete view into the savings provided by fully managed services like Confluent Cloud.

Confluent Cloud now supports Azure Private Link for secure network connectivity, allowing one-way secure connection access from an Azure Virtual Network (VNet) to both Azure and third-party services like Confluent's platform. This feature enables enterprises to address network attack vectors exposed to the public internet by blocking internet access to key resources like Kafka clusters. Unlike VNet peering, which requires coordination on a unique IP address block and can be challenging, Private Link provides network-level security without these downsides. Confluent has adapted its networking stack to run on Azure and integrated it with Azure Standard Load Balancers and Azure Private Link Service, allowing users to spin up Dedicated clusters in minutes directly through the Confluent UI. This feature provides enhanced user experience with secure and hassle-free connectivity, making it ideal for security-conscious companies and highly-regulated organizations.