July 2020 Summaries

ksqlDB 0.10 introduces major updates to key management, enhancing support for multiple key columns and formats like Avro, JSON, and Protobuf. In this version, both tables and streams in ksqlDB manage keys uniquely: tables have a PRIMARY KEY for unique row identification, while streams can declare a KEY column without inferring uniqueness. This update removes the confusing WITH(KEY) syntax, allowing users to name key columns freely and eliminating the need to duplicate keys in Kafka record values. Streams are now supported without key columns, though this may necessitate data repartitioning for operations like joins and aggregations. The new AS_VALUE function allows users to replicate the previous behavior of copying keys into values when necessary. Key column naming has been refined, and users must now explicitly include key columns in projections, impacting materialized views and joins. These changes, while not backward compatible, aim to provide a more intuitive and expressive framework for managing data in Kafka.

Change Data Capture (CDC) is a powerful method for incorporating streaming analytics into existing databases, and Debezium facilitates this process by sending change data through Apache Kafka. This approach is particularly useful when dealing with systems where understanding the changes themselves is analytically valuable, such as when tracking price changes of items in a MongoDB collection. Utilizing Debezium's MongoDB CDC Connector allows for efficient management of record changes by emitting them into a Kafka topic. By leveraging Kafka Streams, these changes are accumulated into a table and then emitted as a new stream of complete records, ensuring consumers can access fully updated data without needing to maintain their own document state or merge logic. This integration is achieved through a combination of Kafka Streams' abstractions and Debezium's metadata, which allows users to access not only before and after versions of data but also deltas, providing flexible options for data consumption. The system can be explored further through a demo environment that showcases the integration's capabilities, offering insights into how Debezium and Kafka Streams can enrich change-only data with historical document states.

Event streaming technology offers a myriad of benefits for big data organizations, but its value depends on the organization's level of maturity in adopting an event-driven architecture. Early use cases provide some value, but full adoption leads to transformational value. As organizations progress from simple pub/sub messaging to connected ecosystems, they begin to reuse data streams across applications and adopt a broader perspective on event streaming. Platform effects amplify the value as the business relies on the platform for mission-critical use cases, leading to massive efficiencies of scale and network effects. Finally, mature organizations create a unified streaming platform that acts as a central nervous system, driving exponential growth in value and enabling new business models. Event streaming provides immediate, reliable, accurate, and intelligent data analysis, and its importance is amplified by the economic climate and the need for reacting to customer events and operational events in near real-time.

PushOwl, a B2B SaaS solution for e-commerce marketing, faced challenges in efficiently managing and retrieving data related to web push notifications, which are critical for their campaign reporting system used by over 22,000 businesses. Initially, PushOwl relied on a PostgreSQL database to store push notifications and generate campaign reports, but as data volume increased, this approach resulted in slow query performance and user experience issues. To address these problems, PushOwl transitioned to an event streaming solution using Apache Kafka and ksqlDB, which allowed for more efficient data handling and scalable stream aggregation. By integrating ksqlDB into their tech stack, PushOwl could leverage Kafka's event streaming capabilities to manage high data throughput and ensure timely updates to their PostgreSQL database. The solution also involved using Kafka Connect to store aggregated event data in cloud storage for further analysis and debugging, ultimately enhancing the performance and scalability of PushOwl's marketing dashboard.

Managing a data center involves complex tasks, including monitoring electrical power, cooling systems, and network bandwidth, which are crucial for maintaining smooth operations. The blog post discusses the role of automated monitoring in this context, highlighting the use of tools like Logstash, Apache Kafka, and Elasticsearch to streamline data collection and alert systems. Moreover, it introduces ksqlDB as a solution for integrating and enriching data from different sources, such as power consumption readings and customer records, through user-defined functions (UDFs) and user-defined table functions (UDTFs). By employing ksqlDB, data can be transformed and enriched upstream, ensuring only the final version reaches the database, thus enhancing efficiency and data privacy. The post details a practical example of using ksqlDB to merge and enrich data from a smart electrical panel and customer information, demonstrating how this approach can improve data visibility for customers while safeguarding privacy. This system ultimately frees personnel from monotonous tasks, allowing them to focus on more meaningful activities within the data center.

You can now easily subscribe to Confluent Cloud using your existing AWS account, eliminating the need for a separate bill. This allows developers to access a managed Apache Kafka service that frees them from operational complexities, while also providing a complete event streaming platform with tools like Confluent Schema Registry and ksqlDB. With Confluent Cloud on AWS Marketplace, developers can simplify their use cases by streamlining data processing and curation, reducing complexity compared to implementing similar solutions using other services. The service is available in two options: Annual Commits and Pay As You Go (PAYG), allowing users to choose the billing method that best suits their needs. To get started, simply search for Confluent Cloud on the AWS Marketplace, select the subscription option, and follow the straightforward subscribing flow. Once subscribed, users can create clusters, set up their account, and start using the service to build event streaming applications with ease.

The need for data privacy in Apache Kafka arises as companies increasingly utilize data for competitive advantage, facing regulatory and reputational risks associated with handling sensitive information. Apache Kafka, a real-time data streaming platform, lacks built-in de-identification protocols compliant with regulations like GDPR and CCPA. The integration of the Privitar Data Privacy Platform with the Confluent Platform via the Privitar Kafka Connector provides a solution, enabling data privacy protections while maintaining analytical utility. This integration allows for the application of Privacy Policies that de-identify sensitive data in Kafka streams, ensuring compliance without the complexities of managing scripts. Privitar's tools allow data scientists to work with privacy-protected data, maintaining statistical relevance and the option for controlled re-identification. This solution is particularly valuable for organizations aiming to manage data privacy across large deployments, offering centralized policy deployment and compliance traceability.

The Confluent Cloud ksqlDB team has developed a comprehensive monitoring and alerting system to address the challenges of managing stream processing services in cloud environments, characterized by the complexity of container orchestration and dependencies on cloud service providers. This system includes automated alerts for potential issues, specific alerts to facilitate prompt resolutions, and secondary metrics monitoring like memory and CPU usage to anticipate problems. The team distinguishes between system and user errors to ensure relevant alerts are actionable, and the alerting system itself features redundancy to detect failures within its pipeline. Iterations of this system were conducted before the launch of Confluent Cloud ksqlDB, emphasizing the need for fine-grained metrics to diagnose provisioning issues and avoid alert fatigue by reducing correlated alerts to a single notification. Collecting metrics about alerts and provisioning processes helps in continuous improvement, informing both internal processes and user expectations.

Kafka Summit 2020, the first virtual edition of this event, is set to take place on August 24th and 25th, offering a comprehensive program led by experts in the Apache Kafka community. The summit will feature eight keynote speakers, including notable figures like Sam Newman and Jay Kreps, and will host 52 breakout sessions and 12 lightning talks covering a wide range of topics suitable for both newcomers and experienced Kafka users. Attendees can engage in live Q&A, participate in Birds of a Feather sessions, and network with over 15,000 Kafka enthusiasts worldwide. The event also offers opportunities to interact with sponsors and experts, and includes pre-summit sessions such as the Apache Kafka Fundamentals course and a Certification Bootcamp, with discounts available on training courses and certification exams. Registration for the summit is free, aiming to build a strong community connection despite the virtual format.

Apache Kafka is a distributed commit log used as a multi-tenant data hub to connect diverse source systems and sink systems, commonly transforming ETL jobs from batch mode to near-real-time mode. It's increasingly becoming the de facto event streaming platform for enterprises across all verticals, democratizing data for both internal and external users or applications of the data. Kafka is making a huge difference in industries like transportation assets, where it can be used to track the movement of assets in real time, specifically for trams, buses, and high-speed electric trains used in the Helsinki Region Transport (HSL) system. The MQTT source produces data into a Kafka topic called vehicle-positions, which is then enriched using Kafka Streams, making it easy for Elasticsearch to consume and display on a dashboard in Kibana. This entire workflow can be accomplished with minimal effort on the development side. After enrichment, the data is written into the topic vehicle-positions-enriched, where it's pushed into Elasticsearch using a sink connector, setting up a dynamic template to recognize the geolocation data. The resulting real-time dashboard running on Kibana provides insights and visualizations of the geolocation data, making it possible for companies to be less reactive and more proactive about meeting the demands of tomorrow starting today.

The blog post discusses the integration of MQTT with Apache Kafka® and Confluent Cloud for real-time interoperability in IoT devices. It explores various use cases across industries such as automotive, manufacturing, energy, oil and gas, logistics, etc., where Kafka and its ecosystem are central to deployments. The post also delves into the pros and cons of MQTT and Kafka, highlighting their complementary nature that makes them a popular combination in IoT projects. It introduces Waterstream, a new option for combining MQTT and Kafka, which can turn your Kafka cluster into a full-featured MQTT broker. The post concludes with a live demo of Waterstream connecting thousands of virtual devices and integrating it with Confluent Cloud and ksqlDB.

The text discusses the different ways to store several event types in the same Apache Kafka topic, specifically when using Confluent Schema Registry. The newer subject-name strategies use record names or topic names to determine schema lookups, but lose subject-topic constraints. In contrast, using a union (or oneof) with schema references maintains subject-topic constraints and allows event types to evolve independently, while also enabling automatic registration of top-level schemas in the case of Protobuf and JSON Schema formats. This approach is particularly useful for querying topics with ksqlDB, as it provides a bounded set of event types defined by the union. Additionally, Confluent Platform 7.7 has been released with enhanced security features, including OAuth support, as well as new connectors and other key features.

Confluent has announced the preview release of its fully managed Elasticsearch Service Sink Connector in Confluent Cloud, an event streaming service based on Apache Kafka. This connector simplifies operations by eliminating the need for users to manage their own Kafka Connect clusters and supports major cloud providers like AWS, Azure, and GCP. Elasticsearch Service, part of Elastic Cloud's SaaS offerings, allows easy deployment and scaling of Elastic products, and it must be located in the same region as the Kafka cluster to avoid data movement charges. The blog details a use case involving a food delivery scenario where new user information is propagated to an Elasticsearch Service index via a Kafka topic, demonstrating the integration process with a Python script and configuration in Confluent Cloud. Users can visualize data with a Kibana dashboard, and the blog also highlights the release of Confluent Platform 7.7, featuring enhanced security and support for Apache Flink and new connectors.

The text introduces the latest developments in Confluent's offerings, focusing on the introduction of infinite storage for Apache Kafka within Confluent Cloud, aiming to eliminate storage limits and enhance scalability. This new feature allows for seamless scaling, removing the need for pre-provisioning storage and enabling enterprises to handle vast amounts of data efficiently. By transforming Kafka into a potential system of record, businesses can streamline operations by accessing both real-time and historical data from a single source, reducing complexity and improving data integration across systems. The blog also highlights the rollout of new features in Confluent Platform 7.7, such as enhanced security measures, support for Apache Flink, and integration capabilities with Amazon OpenSearch, aiming to improve overall performance and cost-effectiveness for users.