February 2021 Summaries

Kerberos authentication is a key component in many client/server applications, but setting it up can be challenging without prior experience, especially for SSH-based systems. This tutorial from Confluent provides a detailed guide to creating a Kerberos test environment for passwordless authentication using containers, which are beneficial for their speed and iterative capabilities. The setup involves configuring a Kerberos Key Distribution Center (KDC) server, an SSH server, and a client, all using OpenSSH and MIT Kerberos software. The tutorial outlines two approaches: a triple container setup with separate client and server containers and a single-container setup that combines the KDC and SSH server, allowing the host machine to act as the client. Detailed steps include adding principals, configuring keytab files, and ensuring proper communication between the server and client. The guide is tailored for Mac users but notes adjustments might be needed for other systems, and it provides troubleshooting tips and suggestions for further exploration with Confluent's connectors.

The blog post provides a comprehensive guide on deploying the Confluent Platform on Kubernetes using the Confluent Operator, with a focus on secure credential management through HashiCorp’s Vault. It details an end-to-end demo that involves setting up a connector which monitors commits to a GitHub repository and writes the commit messages to an Apache Kafka topic. The demo emphasizes the importance of securely storing and managing credentials, highlighting the risks associated with traditional methods of credential distribution. The text describes the step-by-step process of deploying a Google Kubernetes Engine cluster, initializing Vault, configuring policies, and integrating the GitHub connector, ensuring that sensitive credentials are not exposed in plaintext. The demonstration showcases a seamless integration between Confluent Operator and Vault, allowing credentials to be securely accessed in-memory by the necessary components during runtime. The post concludes by suggesting that the demonstrated pattern for secret management can be applied broadly across various connectors and components within the Confluent Platform, promoting best practices for secure and efficient DevOps in multi-environment deployments.

The ODP Source Connector is a fully supported, turnkey solution for integrating SAP systems with Apache Kafka, leveraging Kafka Connect and Confluent Platform. It provides high scalability and performance, simple installation and operations, and mission-critical support SLAs. The connector integrates seamlessly with Kafka Connect and Confluent Platform, offers tight integration with the Kafka ecosystem via the Kafka Connect API, and supports delta-enabled ODP data sources for exactly-once delivery guarantees. It is designed to be configured and started once and run forever without complex administration tasks. The connector is a cost-effective, Kafka-native, lightweight, and scalable solution for consuming business information out of SAP AS ABAP-based systems in Kafka, suitable for real-time messaging, data integration, and data processing at scale.

Apache Kafka plays a critical role in Pinterest's data transportation layer, handling a rapidly increasing data flow due to the platform's significant growth in user base and content. This growth has led to various operational challenges, such as performance issues with magnetic disks, which were resolved by switching to SSDs for better IOPS and reduced latency. Pinterest has implemented dynamic rebalancing strategies and message format conversions to optimize Kafka's performance, alongside cost control measures like compression and rack-aware data transfer to reduce AWS costs. The introduction of brokersets has improved topic scaling and partition placement, minimizing the operational impact of traffic spikes. To manage Kafka clusters more efficiently, Pinterest developed Orion, a unified management tool with features that enhance automation and provide better visibility and control over cluster operations. Recent upgrades have focused on ensuring consistency in broker versions and log message formats, with due diligence and client compatibility checks to minimize disruptions. Looking ahead, Pinterest is working on improving client interoperability, scaling efficiency, and enhancing the reliability of its Kafka infrastructure.

The latest release of ksqlDB, version 0.15, introduces significant enhancements, particularly in its handling of message keys in Apache Kafka. Previously limited in the types of data it could process from message keys, ksqlDB now supports a broader range, including multiple columns and advanced data types like ARRAY and STRUCT, across all supported serialization formats such as JSON, AVRO, and PROTOBUF. This update allows for more complex operations, including the use of PARTITION BY and GROUP BY clauses with multiple expressions, resulting in streams and tables with multiple key columns. Despite these advancements, certain limitations persist, such as the inability to support JOIN operations on sources with multiple key columns, although workarounds involving STRUCT-type key columns are possible. The update also improves integration with Confluent Schema Registry, enabling automatic schema inference for both key and value columns. Future developments promise further enhancements in handling multiple key columns and more sophisticated data operations, positioning ksqlDB as a more robust tool for managing and querying streaming data in Kafka environments.

Apache Kafka is moving away from its dependency on ZooKeeper, a distributed configuration service, with the introduction of KIP-500, which includes a built-in consensus layer in Apache Kafka 2.8.0. This change simplifies infrastructure design and operational workflows for Kafka deployments by removing the need to administer an additional system. By removing ZooKeeper, users can stop worrying about learning another distributed system, administering separate servers or VMs, and managing security configurations. Additionally, this change reduces storage considerations, as Kafka no longer requires capacity planning for ZooKeeper. The new consensus layer in KIP-500 improves control plane traffic, reducing the time it takes to read metadata from ZooKeeper and improving operations. This change also eliminates the need to monitor ZooKeeper metrics and troubleshoot issues related to divergent state between the controller and ZooKeeper. Users can now focus on core components of Kafka without the added complexity of managing ZooKeeper.

ksqlDB 0.15 introduces significant enhancements, including expanded support for complex message keys and improvements to the command line interface for interactive development. This release allows users to utilize a wider range of data types in message keys, such as JSON, AVRO, and PROTOBUF, as well as structured data types like ARRAY and STRUCT, thereby unlocking data that was previously inaccessible. The update also facilitates the use of PARTITION BY and GROUP BY clauses with multiple expressions, enabling the creation of tables and streams with multiple key columns. Additionally, the process for terminating persistent queries has been simplified by allowing a DROP command to automatically terminate associated queries, eliminating the need to know query IDs in advance. These enhancements make it easier to manage streams and tables, and users are encouraged to explore the full range of features by utilizing ksqlDB through Confluent Cloud or standalone distributions.

Confluent has released the Oracle CDC Source Connector, which allows organizations to reliably and cost-effectively implement continuous real-time syncs by offloading data from Oracle Database to Confluent. This connector can be used alongside other Confluent components like ksqlDB and sink connectors for modern data systems to enable key use cases such as data synchronization, real-time analytics, and data warehouse modernization. The connector uses Oracle LogMiner to read the database's redo log and requires supplemental logging with ALL columns for tables that are of interest or for the entire database. It supports multiple versions of Oracle Database and can start reading logs from a specific system change number (SCN) or timestamp. Confluent's Oracle CDC Source Connector can continuously monitor the original database and create an event stream in the cloud with a full snapshot of all of the original data and all of the subsequent changes to data in the database, as they occur and in the same order. The connector can be used to write change events specific to tables in the Oracle Database to Kafka topics for further processing or analysis. Confluent has also released a fully managed BigQuery Sink Connector that can consume these event streams and apply those same changes to the BigQuery data warehouse, providing a working data pipeline from an Oracle database through to BigQuery.

The introduction of Automatic Observer Promotion in Confluent Platform 6.1 allows for a fast, automatic, and no-data-loss option for maintaining topic availability in the event of a failure in multi-datacenter stretch clusters. This feature is part of Multi-Region Clusters built directly into Confluent, which enables running a single Apache Kafka cluster across multiple datacenters with synchronous replication. Observers are effectively asynchronous replicas that replicate partitions from the leader but can never participate in the ISR or become a partition leader unless manually promoted through unclean leader election. Automatic Observer Promotion automates this process by promoting caught-up observers to the ISR list when a replica fails, restoring availability and allowing producers to once again produce data. This feature eliminates the need for manual intervention, reduces operational burden, and minimizes the risk of data loss and partition unavailability due to failover. It provides an RPO of zero while maintaining a low RTO, making it an essential tool for modern businesses that require mission-critical data protection.

Confluent Platform 6.1 offers several key features to simplify management tasks for Apache Kafka operators, including Automatic Observer Promotion, which automates the process of promoting observer replicas in case of a datacenter failure, reducing downtime and business disruption. The release also introduces enhancements to Multi-Region Clusters (MRC), allowing for even higher availability for enterprises using Kafka as their central nervous system and data backbone for business-critical applications. Additionally, Confluent Platform 6.1 supports in-place upgrades of ksqlDB production queries, enabling developers to iterate and make improvements to their mission-critical applications without sacrificing availability. The release also expands support for multi-key pull queries in ksqlDB, allowing for more flexible lookups using multiple keys, and integrates Cluster Registry with Confluent Control Center, providing a centralized, consistent, and easy-to-remember cluster naming experience. Furthermore, Confluent Platform 6.1 introduces a Status API that tracks the progress of Self-Balancing Clusters, enabling operators to better control their clusters and ensure healthy operations for mission-critical applications.

Kafka Connect, a component of Apache Kafka, facilitates the streaming integration of external systems into and out of Kafka using connectors, which can be either pre-existing or custom-developed through the Kafka Connect framework. This framework simplifies the process of writing source connectors to fetch data from external sources and return it as SourceRecord instances, or sink connectors to write Kafka data to external systems, without needing direct interaction with Kafka. Configuring these connectors is critical and requires defining parameters such as system location and authentication details, which are validated for accuracy to prevent errors during deployment. The Kafka Connect framework offers a ConfigDef API to define these parameters, allowing for the use of validators to ensure values are correct and recommenders to suggest valid options based on interdependent configurations. If values depend on other configurations, developers may need to manually override the validate() method to ensure comprehensive validation. This process is crucial for deploying connectors effectively, and additional resources are available through Confluent Developer for those interested in further exploring Kafka and event streaming.

A workshop on building predictive machine learning with Flink will be held on December 18, and registration is now open. The Apache Kafka-based architecture naturally decouples systems, allowing the best tool for the job to be used. A JavaScript client called KafkaJS has been developed, which provides a lightweight approach suitable for edge deployments or serverless functions. The project began as an independent initiative and has gained traction with 100,000 downloads per week and a vibrant community. It is also used by other frameworks such as NestJS and n8n. A tutorial on building a Slack bot that notifies users of new package releases published to the Node Package Registry will be provided. The application utilizes npm-hook-receiver for receiving webhook requests and publishing messages to Kafka. The consumer's job is to consume messages from the topic and post notifications to Slack. The application can be run locally or in Confluent Cloud, with support available through GitHub sponsors.

The blog post provides a detailed tutorial on building real-time data pipelines using Confluent Cloud and Azure Databricks within Microsoft Azure, highlighting how these platforms simplify processing IoT, change data capture, and streaming data without the need for complex infrastructure setups. It guides users through configuring Azure Databricks to interact with Confluent Cloud, ingesting and processing data from Apache Kafka topics, and utilizing a secured Confluent Schema Registry with AVRO data format. The step-by-step instructions cover setting up clusters, managing API keys, creating Kafka topics, and using Python code in Databricks for data manipulation and storage on Azure Data Lake Storage in Delta Lake. Additionally, the post emphasizes the importance of schema management and provides insights into using Spark Structured Streaming for continuous data processing, enabling users to build scalable and efficient data pipelines for real-time business insights. The blog also highlights promotional offers for new Confluent Cloud users and announces the general availability of Confluent Platform 7.7 with enhanced security features and integrations.

Confluent has launched the Confluent Community Forum on the Discourse platform, offering a new space for community members to ask questions, share best practices, and connect. While the Confluent Community Slack remains a key component for real-time interactions, the forum is designed for more permanent, searchable content and in-depth discussions. Users can sign up with an email or GitHub account, navigate various categories, and participate in discussions with rich formatting using Markdown. The forum encourages participation through a system of badges and flairs, rewarding users for contributions and milestones. The Confluent Community Forum complements existing communication channels, providing an additional option for engagement alongside Slack, meetups, and mailing lists.