April 2020 Summaries

Bincover is a tool that measures the coverage of Go binary applications. It provides a simple and flexible API to generate an "instrumented binary" that can measure its own coverage, run it with user-specified command line arguments and environment variables, and merge coverage profiles generated from multiple test runs. The main idea behind measuring Go binary coverage is to create a file with a "fake" test that runs the main method of the application, but this approach has limitations such as not being able to pass in command line arguments or handling panics. Bincover handles these problems while requiring no additional lines of code to do so. It can be used to measure the coverage of actual Go applications by creating a temporary args file and setting an -args-file flag, parsing the arguments, executing the binary, truncating and deleting the file, and merging temporary coverage files into a single file. Bincover is now available on GitHub and Confluent Platform 7.9 has been released with its latest key features including Oracle XStream CDC Connector, Client-Side Field Level Encryption, Confluent for VS Code, and more.

In response to the COVID-19 pandemic, the Kafka Summit planned for Austin, TX, was transformed into a virtual event titled "Kafka Summit 2020: Event Streaming Everywhere," allowing global participation in an online format. This shift prioritizes the safety of attendees, sponsors, and the community while still facilitating the sharing of best practices and future discussions among developers, data architects, engineers, and DevOps professionals involved in event streaming platforms like Kafka. The virtual summit is free to attend, and refunds are being issued to those who paid for the originally planned in-person event. The event is scheduled for August 24th-25th, with an open call for papers until May 17th for those interested in speaking. Further details about the virtual experience will be released soon, and participants are encouraged to sign up for updates.

Mainframes remain integral to global financial transactions, yet modern technologies such as Kafka, ksqlDB, and Apache Cassandra are revitalizing mainframe operations at Broadcom. The Mainframe Operational Intelligence (MOI) product utilizes these technologies to collect and analyze performance metrics from mainframes, addressing challenges like high metric variability and false-positive alerts. A prototype using ksqlDB demonstrates its potential as a filtering tool for anomaly detection by employing SQL-like rules to manage streams of metrics. This involves creating streams with specific rules to filter and flag alerts based on metric values. Despite challenges such as schema compatibility, the integration of ksqlDB into MOI has shown promising results in enhancing the system's efficiency. The effort reflects a broader trend of integrating innovative data processing tools to improve legacy systems, supported by a responsive community and comprehensive resources, as evidenced by Skai's success in overhauling its ad-campaign dashboard using similar technologies.

Confluent Schema Registry has been enhanced to support Protocol Buffers (Protobuf) and JSON Schema formats in addition to Avro, allowing users to choose from the most popular formats and add new ones as needed. The platform now supports multiple formats at the same time, schema references, and compatibility rules, enabling backward-compatible or forward-compatible evolution of Protobuf schemas. New serializers and deserializers are available for Protobuf and JSON Schema, along with command line tools and Confluent CLI options, making it easy to get started with these new formats. Kafka Connect converters have been added for Protobuf and JSON Schema, as well as support in Confluent REST Proxy and Kafka Streams. Users of ksqlDB can now specify VALUE_FORMAT='PROTOBUF' or VALUE_FORMAT='JSON_SR', and Confluent Control Center offers visualization and schema creation capabilities for Protobuf and JSON Schema topics. The platform is being rolled out to Confluent Cloud, and users are free to add new formats and define compatibility rules using the extensible architecture of Schema Registry.

Confluent has released version 5.5 of its Confluent Platform, which enhances three categories of features that complement Apache Kafka® and make it easier to build event streaming applications. The release includes support for Protobuf and JSON throughout its components, expanding the choice of serialization formats. Additionally, Confluent Platform 5.5 introduces exactly-once semantics to librdkafka-based clients by adding support for idempotent producer and transactions. This allows applications written in C/C++, Python, Go, or .NET to guarantee that messages processed by a given event streaming application will be processed exactly once. The release also adds several admin functions to the REST Proxy in preview mode and includes ksqlDB 0.7 with new features for building event streaming applications. Confluent Platform 5.5 is built on Apache Kafka version 2.5, which adds several new features such as OAuth support and a new Connector.

Confluent, a company specializing in event streaming, has raised $250 million in its Series E funding round, valuing the company at $4.5 billion. The investment was led by Coatue Management with participation from Altimeter Capital and Franklin Templeton, as well as existing investors Index Ventures and Sequoia Capital. Confluent's cloud offering, Confluent Cloud, has seen significant growth in adoption and revenue. Event streaming is emerging as a major new technology category that could be as important and foundational to modern businesses as databases have been. The company plans to accelerate its efforts with the launch of Project Metamorphosis, which will introduce new product capabilities over the next eight months. Confluent has also been named Microsoft's 2024 OSS on Azure Global Partner of the Year for its commitment to delivering outstanding open source-based applications and infrastructure solutions on Microsoft Azure.

The text provides a detailed tutorial on implementing a real-time data push from Apache Kafka to a web application using Confluent Platform and JavaScript. The guide emphasizes the demand for faster data delivery to business users, bypassing traditional data pipelines by directly pushing events as they occur. It outlines the steps to set up an architecture using minimal JavaScript code and a single-node Kafka cluster, employing the Confluent CLI and HTTP sink connector for seamless data flow. The tutorial also includes a practical demonstration using a Node.js web service to forward Kafka messages as Server-Sent Events (SSE) to the frontend, ensuring that the system efficiently handles events with visual confirmations in a web browser. Additionally, it touches upon using Tableflow to transform Kafka topics into queryable Iceberg tables, leveraging UK Environment Agency sensor data, and discusses improvements in Kafka consumer offsets management through KIP-1094 for enhanced data processing reliability and performance.

The Apache Kafka community has released version 2.5.0, which includes several new features and improvements, such as the simplification of APIs for transactional use cases, improved producer state retention, ZooKeeper TLS support, and enhanced topic tracking. The release also introduces a new co-group operator to simplify aggregating multiple streams into one table, a toTable() function to materialize tables over streams, and improved support for interactive queries against state stores. Additionally, the community has dropped support for Scala 2.11 in this version and set TLS 1.2 as the default SSL protocol.

The GenAI Streamposium discusses the potential of event-driven stream processing for small to medium-sized businesses. Confluent's ksqlDB is introduced as a tool that allows organizations of any size to build event streaming pipelines with minimal infrastructure and code, using a language intuitive to relational databases. The use case presented involves an online pharmacy that uses ksqlDB to process its transactional data, transforming it into valuable insights on the impact of discounting on average order value. By leveraging existing data sources and expertise in Kafka, the company gains real-time feedback and a set of tools capable of transforming data without specific visualization or storage engine requirements. This approach demonstrates how improved business intelligence can be achieved by small to medium-sized businesses, regardless of their industry or size, using ksqlDB as an agile and adaptable framework.

This webinar discusses how to build GenAI apps faster using Kafka Streams' real-time stream processing technology. The main goal is to detect compromised accounts and malicious user behavior by analyzing website login attempts. The analysis involves four types of analysis: device recognition, location recognition, botnet detection, and brute-force attack detection. The system uses Apache Kafka as the event streaming platform and Kafka Streams API for stateless and stateful processing operations. The solution is designed to be scalable, fault-tolerant, and efficient in terms of memory usage. The final design optimizes the pipeline by using a composite key for the join operator, eliminating the need for a stateful aggregate operator.

Confluent Hub has undergone a frontend redesign, featuring a detailed plugin taxonomy that allows users to filter plugins across various dimensions such as enterprise support type and performance stability verification levels. The platform now offers over 140 plugins, including fully supported ones by Confluent and its partners. Users can filter plugins based on their development environment requirements, such as enterprise scale or high-performance stability. Additionally, Confluent Hub provides a new way for developers to discover and share useful code, making it easier for users to find the necessary plugins for their projects.

Confluent Cloud KSQL has transitioned from private preview to general availability, providing a production-ready solution for users on usage-based billing plans. This development enhances data stream processing by leveraging Apache Kafka as a central data source and simplifying complex operations like transformations and aggregations using SQL syntax, eliminating the need for intricate application code. Confluent Cloud KSQL offers a fully managed service that abstracts infrastructure complexities, allowing organizations to focus on utilizing data for business solutions. It features an intuitive web interface for stream and query management, a "Flow" view for visualizing processing workloads, and seamless integration with other Confluent Cloud components like Kafka and Schema Registry. Users can also connect programmatically via the REST endpoint, ensuring real-time intelligence delivery with minimal operational overhead. To facilitate user onboarding, Confluent Cloud offers a $50 monthly credit for the first three months.

The text discusses the challenges of running a managed cloud service for Apache Kafka, a popular open-source messaging system. The author, from Confluent, shares their experiences and strategies for ensuring reliability, stability, and performance in a cloud-based environment. They explain how they implement rigorous testing processes, including automated tests, soak tests, and nightly performance runs, to catch regressions and ensure the quality of their service. The article also touches on the importance of continuous integration and deployment (CI/CD) pipelines, automated testing frameworks, and monitoring systems to detect issues early on. Additionally, it highlights Confluent's efforts to improve their testing efforts by introducing automated performance regression tests, a user-friendly platform for benchmarking and reporting, and splitting monolithic releases into independent components. Overall, the article provides insights into the challenges of running a cloud-based Kafka service and how Confluent addresses these challenges through their testing and deployment practices.