September 2020 Summaries

The ksqlDB 0.12.0 release introduces significant improvements to enhance usability and reduce administration time, featuring query upgrades and automatic restarts for persistent queries. Users can now evolve queries in real-time with the "CREATE OR REPLACE" feature, allowing for seamless modification without disrupting stream processing, as seen in the example of a fictional ksqlMart store updating its return policy. Schema upgrades are also supported, enabling the addition of new fields to streams while maintaining query continuity. Additionally, persistent queries that encounter errors now automatically restart, resuming from the last processed event, which minimizes operational disruptions and saves resources. These enhancements, along with other updates detailed in the changelog, are aimed at optimizing the experience of working with ksqlDB, with further information available on GitHub and the community Slack channel.

The ksqlDB system is an event streaming database that uses Apache Kafka as its underlying data store. It provides a SQL-like interface for stream processing, allowing users to model events as rows and insert them into streams. Streams are topics with a strongly defined schema, and each row in the stream has additional metadata. The system uses persistent queries to transform one stream into another, which run indefinitely and manage their own processing progression. ksqlDB handles data locality by partitioning data across brokers, making it easier to process smaller slices of data. It also inherits Kafka's conflict-free consumption model, allowing multiple consumers to read from the same topic in a shared manner. The system is designed to be composable, concise, and elegant, offering a higher-productivity interface for working with Kafka without diluting its core concepts.

Twitter, a leading social media platform, has revamped its recommendation systems by implementing a new streaming data logging pipeline for its home timeline prediction system, utilizing Apache Kafka® and Kafka Streams, in order to handle billions of tweets daily. This upgrade, which replaces an older offline batch system, significantly reduces pipeline latency from seven days to one day, improving model quality and engineering efficiency. Central to this system is a customized left-join functionality in Kafka Streams that efficiently matches features and labels in machine learning models, allowing Twitter to maintain up-to-date models that adapt to changing user behaviors and trends. The blog post details this customization process, highlighting the unique challenges and solutions, such as handling consumer lag and ensuring data quality, while also acknowledging the contributions of numerous team members and the potential future enhancement of cooperative rebalancing to further bolster the pipeline's performance.

The streaming-ops project utilizes Kubernetes, GitOps, Confluent Cloud, and Kubernetes Operators to manage complex event-driven systems on top of Apache Kafka. It demonstrates the use of declarative resource management, secret management, and automated deployment using Flux, Kustomize, and Bitnami Sealed Secrets. The project also showcases the management of Kafka Connect workers and connectors, as well as the integration with Confluent Cloud services. By leveraging Kubernetes Operators, it provides a scalable and secure way to manage these resources and ensure accurate automation.

Google Anthos offers the benefits of cloud computing in datacenters, other clouds, and at the edge, allowing modernization of applications and infrastructure without migrating to the cloud. Infrastructure management, container management, service management, and policy management across environments provide a strong foundation for data infrastructure and applications. Confluent unifies data infrastructure on top of Anthos, enabling consistent data access across environments. Event-driven architectures are an increasingly popular strategy for decoupling microservices, with Kafka being used as the data plane across multiple environments. Deploying across edge sites and providing a consistent environment is important when managing hybrid architectures. Combining Anthos and Kafka sets the stage for a symphony of databases, applications, and data processing systems in a hybrid future.

The guide provides a detailed overview of setting up secure connections between MongoDB Atlas and Confluent Cloud using VPC peering and PrivateLink. It explains the necessary steps to configure a VPC-peered Apache Kafka cluster in Confluent Cloud with an AWS VPC, highlighting the need to avoid using open IP ranges due to security concerns. The document outlines the requirements for creating such connections, including having a dedicated Kafka cluster and a MongoDB Atlas cluster with AWS, and describes how to establish a PrivateLink between AWS and MongoDB Atlas for enhanced security. The process involves configuring network settings, accepting VPC peering requests, and setting up route tables to enable seamless data connectivity. Additionally, the guide mentions that MongoDB Atlas source connectors only support username and password for database access, and stresses the importance of ensuring the MongoDB Atlas database is in the same region as the Kafka cluster to avoid data movement charges. It concludes by promoting Confluent Cloud as a fully managed event streaming service and offers a promotional code for free usage credits.

Azure Data Explorer, also known as Kusto, is a big data analytical database service designed for near-real-time insights into scalable log analytics, time series analytics, and IoT/telemetry analytics on Azure. The blog post outlines a process to integrate event streaming from Apache Kafka on Confluent Cloud into Azure Data Explorer using the Kafka Connect Kusto Sink Connector. Kafka Connect is an open-source, distributed integration service that simplifies data streaming between sources and sinks without requiring code, supporting features such as fault tolerance and per-record transformation. The post provides detailed instructions for setting up a streaming pipeline, including creating a Confluent Cloud instance, configuring Azure Data Explorer, and deploying Kafka Connect on Azure Kubernetes Service. It also highlights the use of Docker and Helm for managing connector infrastructure and encourages participation in the open-source project, offering a promo code for additional Confluent Cloud usage.

Confluent Platform has been validated for integration with AWS Outposts, a fully managed service that allows users to run instances as they would on Amazon EC2 locally in their datacenter or co-location facility. This enables a true hybrid experience with a unified control plane for AWS infrastructure and a unified event streaming platform to build out next-generation event-driven applications. The advantages of running Confluent Platform on AWS Outposts include overcoming challenges due to data locality requirements and running complex hybrid scenarios between on-premise and AWS services. To deploy Confluent Platform on AWS Outposts, users need to create a custom Amazon Machine Image (AMI) on top of Ubuntu Linux, configure CloudFormation templates, and set up connectivity with EC2 instances, Kafka brokers, and the internet. The deployment process involves creating an Ansible inventory node, configuring cloud-init, and running Kafka topics and avro console producers and consumers to test the integration.

Apache Kafka, an event streaming platform, does not inherently support message prioritization, a common feature in traditional messaging systems like JMS and AMQP, due to its architectural design focused on immutable data streams and efficient data processing. Kafka's architecture revolves around a commit log, ensuring data immutability and ordered record processing, which inherently conflicts with the concept of message reordering based on priority. To address the lack of prioritization, a "bucket priority pattern" is proposed, where messages are grouped into different priority buckets determined by custom partitioners and assignors, allowing consumers to process messages according to priority without altering Kafka's core principles. This pattern involves producers and consumers determining bucket allocations and processing messages accordingly, though it introduces complexity in managing partitions and requires careful configuration. Despite these challenges, the bucket priority pattern offers a practical solution for implementing message prioritization within Kafka's event streaming framework, maintaining the platform's strengths in scalability, fault tolerance, and parallelism.

Confluent Operator facilitates the deployment and management of the Confluent Platform as a cloud-native, stateful container application on Kubernetes and OpenShift, simplifying the process of operating Apache Kafka® clusters. By leveraging Kubernetes automation, OpenShift Container Platform (OCP), and Helm, it offers benefits like cloud portability, self-healing capabilities, and easy scalability, while maintaining user control over components. OCP, a platform-as-a-service built around Docker and Kubernetes on Red Hat Enterprise Linux, can be deployed on-premises or in public clouds, although it requires managing certain responsibilities like securing Kubernetes nodes. The setup process involves deploying OCP, followed by the Confluent Operator and its components, with the necessity of cloud provider accounts, Red Hat network access, and familiarity with OCP SDK. Deploying Kafka on Kubernetes via Confluent Operator provides native integration with Prometheus and Grafana for metrics visualization, and the use of StatefulSets for Kafka brokers and other components ensures self-healing capabilities, enhancing the reliability and efficiency of cloud-native deployments.

The Bridge to Azure architecture allows event streaming applications to run anywhere and everywhere using Microsoft Azure, Confluent Replicator, and Confluent Cloud. This enables organizations to migrate or extend any application running on premises or other clouds to Azure. The architecture uses Confluent Platform for data ingestion and processing, Confluent Replicator for real-time replication across environments, and Confluent Cloud for scalable and secure event streaming. Additionally, the architecture integrates with various Azure services such as Azure Data Lake Storage (ADLS), Azure SQL Database, and Power BI, allowing organizations to leverage these services for data analysis and visualization. By using this architecture, organizations can create a digital-first strategy that meets customers where they are, while also ensuring data governance, security, and scalability.

Confluent Cloud provides a simple way to create a serverless Kafka environment for development and testing needs, allowing users to automate their workflow and build automated CI/CD pipelines with the Confluent Cloud CLI and REST APIs. The ccloud-stack utility simplifies the creation of Confluent Cloud services, including provisioning all needed resources and credentials in a single command. Users can create a new Confluent Cloud stack using the ccloud-stack utility or programmatically set up 100% of their Kafka services in the cloud using the Confluent Cloud CLI. The fully managed Datagen Source Connector for Confluent Cloud provides predefined schema definitions with complex records and multiple fields, allowing users to generate interesting test data for their topics. Additionally, users can deploy connectors from external systems without operational overhead, and use the Confluent Cloud UI to visualize their data flowing through their cluster.