November 2022 Summaries

Integrating Confluent Platform with Datadog on AWS's Elastic Kubernetes Service (EKS) allows organizations to enhance the monitoring and management of their Apache Kafka deployments. This setup facilitates collecting comprehensive telemetry data from Kafka, providing insights into the health and performance of real-time data pipelines. By deploying Confluent for Kubernetes (CFK), a cloud-native control plane, users can manage Confluent in private cloud environments using a declarative API. Datadog, an analytics tool, helps visualize Kafka's performance in real time, enabling operations teams to correlate these metrics with other applications and create tailored alerts. Installing Datadog agents across Kubernetes nodes ensures robust monitoring of Kafka clusters, while the integration offers a centralized dashboard to monitor various Kafka components. This comprehensive visibility supports proactive responses, secures data pipelines, and maintains overall system health, making it a preferred solution for many Confluent customers.

Confluent introduced a Thanksgiving-themed Stream Processing Use Case Recipe for real-time survey response analysis using Apache Kafka, ksqlDB, and Confluent Cloud. The recipe explains how survey response and respondent data can be ingested into Kafka through source connectors such as the ServiceNow Source Connector, modeled as streams and tables, protected through respondent-data masking, joined, transformed, and aggregated to generate timely insights. It includes sample data for users without existing survey datasets and emphasizes that rapid analysis helps organizations respond before feedback loses relevance. To demonstrate how the approach can be customized, the author built a Telegram survey bot that consumes questions from a Kafka topic, collects friends’ pie preferences, formats responses with Python, sends them back to Kafka, and enables live aggregation of the results for a Friendsgiving dessert decision.

Kafka Summit London’s open Call for Papers invites Apache Kafka practitioners of all experience levels to submit technical talk proposals by drawing on projects, lessons learned, tools, architectures, ecosystem integrations, feature deep dives, or contributions to the open-source community. Prospective speakers are encouraged to begin with their own experiences, clearly explain the motivation and challenges behind their work, and structure presentations as stories that address why a problem mattered, how it was solved, and what audiences can do with the resulting knowledge. Strong talks should make company-specific experiences broadly useful by emphasizing transferable tips, practical implications, and relevant outcomes for attendees. Feedback from colleagues, community members, and conference office hours can help refine abstracts and align them with program expectations, while speaking offers opportunities to share knowledge, build connections, and contribute to the Kafka community.

Confluent migrated critical Helm-based control-plane and monitoring services for its multi-cloud Kafka platform from monthly, manually coordinated “Big Bang Releases” to an in-house Kubernetes deployment management platform designed to make deployments declarative, auditable, secure, and less error-prone. The platform uses a controller pattern to select target clusters through metadata rules, render Helm charts with Kubernetes and Vault-provided values, stage manifests, and have cluster agents apply them, while providing health-based deployment feedback, Slack notifications, immutable Git-managed service versions, CI integration, and automatic bootstrapping of new clusters. To simplify migration, the team preserved immutable Kubernetes attributes for in-place and potentially zero-downtime transitions, automated YAML generation, reused existing secret formats, and added dry-run comparison tools. Performance work reduced memory consumption from more than 3 GB to about 256 MB, and load testing indicated capacity for ten times the expected new workload. Security improvements replaced a shared Vault authentication role with cluster-specific roles and added RBAC, ultimately enabling migration of all targeted services, reducing memory-related costs by 90%, and improving deployment visibility, while future work focuses on horizontal scalability, authentication challenges, and more complex service targeting needs.

Meshach Cisero, Senior Manager of Growth Marketing at Confluent, describes a career path from consulting, agencies, and hospitality marketing into technology, where he developed SQL and data-analysis skills and sought to contribute to greater racial representation in Silicon Valley. After joining Confluent remotely, he found an inclusive workplace culture, earned a promotion within his first year, and became a leader in Blackfluent, the company’s Black employee resource group. He highlights Blackfluent and its Wakanda subgroup as spaces for Black employees to discuss personal and community issues, build relationships, and find support through activities ranging from professional networking to informal virtual gatherings. Cisero connects his ERG leadership to his experiences as a first-generation Black student at a predominantly white university and to his longstanding involvement in multicultural and Black student organizations. In growth marketing, he has progressed from signup acquisition to product activation, customer research, pricing interviews, onboarding initiatives, and collaboration with product teams, valuing the role’s broad scope and strategic flexibility. He views Confluent’s data-focused products and continued growth as opportunities for further innovation, professional challenge, and advancement.

Confluent promoted its presence as a Diamond sponsor at AWS re:Invent 2022 in Las Vegas from November 28 to December 2, highlighting Confluent Cloud on AWS and Apache Kafka capabilities for managed, cloud-native data streaming. At booth #3535, attendees could attend breakout sessions and live demonstrations focused on connecting multicloud and hybrid pipelines, building real-time applications, reducing data silos, and governing streaming data flows for uses including machine learning and serverless applications. The company also offered meetings with Kafka specialists, $400 in Confluent Cloud credits for qualifying booth sign-ups, and a November 30 Desert Disco networking party hosted with MongoDB at Sugarcane Raw Bar Grill.

Apache Kafka's consumer rebalancing process can be complex and impact performance if not managed properly. Understanding the different types of rebalancing, such as Stop-the-World and Incremental Cooperative rebalancing, is crucial to minimizing downtime and optimizing processing. Monitoring key metrics like join rate, records lag, and memory consumption can help identify potential issues before they become major problems. Additionally, ensuring that consumer groups are properly sized and configured, and that group coordinators are not overwhelmed with too many groups or partitions, can also help reduce rebalancing time and improve overall system performance.

When encountering problems with Apache Kafka, such as an exploding number of connections or wonky record batching, it's essential to consider these issues as symptoms of a broader problem rather than isolated problems. Instead of treating individual symptoms, diagnosing and debugging can help uncover the root cause. Monitoring broker metrics, such as requests per second, network processor idle percentage, request queue size, total time taken for requests, and response send time, can provide valuable insights into the performance of Kafka clusters. By analyzing these metrics, developers can identify potential issues, such as inefficient batching, high client metadata requests, or over-committing consumers, and take corrective action to optimize their Kafka setup. Understanding the root cause of problems is crucial before making changes to applications, and exploring additional resources, such as Kafka Connect and Kafka Streams, can help deepen knowledge of the Kafka ecosystem.

Confluent Developer offers a free hands-on course on securing Apache Kafka workloads with Confluent Cloud, emphasizing that open-source Kafka is unsecured by default and can be difficult to protect and manage when self-hosted. The course covers authentication through SAML-based single sign-on, API keys, and OAuth, as well as authorization through ACLs and scalable role-based access control, which centralizes permissions and can simplify compliance reviews. Confluent Cloud encrypts data at rest and in transit using TLS 1.2, supports bring-your-own-key encryption and private networking options, and captures audit logs by default in Kafka topics for near-real-time monitoring. It also highlights compliance support for standards and regulations including SOC, PCI DSS, ISO 27001, GDPR, CCPA, HIPAA, TiSAX, and financial-services requirements, positioning managed security features as tools for organizations handling sensitive or regulated data.

The text discusses common issues encountered when using Apache Kafka, particularly focusing on diagnosing and addressing inefficiencies in data batching to maintain high throughput. It emphasizes that many apparent problems, such as an increasing number of connections or inconsistent record batching, are often symptoms of deeper configuration issues. The article explains the importance of monitoring Kafka producer metrics, like batch-size-avg and records-per-request-avg, to assess batching efficiency. Key configuration settings such as batch.size, linger.ms, and buffer.memory are highlighted as crucial for optimizing batching, while also considering the impact of topic partitioning and scaling on performance. The text encourages a thorough diagnosis of issues before making any changes, suggesting that an understanding of the underlying causes is essential for effective problem-solving in Kafka applications.

Confluent Cloud is a fully managed, cloud-native Kafka service that provides connectors, Kafka, ksqlDB, Schema Registry, monitoring, governance, security, and enterprise support across AWS, Azure, and Google Cloud through consumption-based pricing. While Confluent manages underlying infrastructure, installation, upgrades, encryption, availability, and throughput guarantees, service partners remain responsible for designing, building, deploying, and supporting customer applications. Developers configure connectors, topics, schemas, data retention, replication, and ksqlDB pipelines; operators monitor environments, manage application deployments and changes, and support applications; and architects define solution topology, capacity, disaster recovery, security, access controls, and topic-level performance strategies. The central distinction is that Confluent provides the managed streaming platform, while partners apply customer-domain knowledge, integration expertise, and application-development capabilities to deliver business value without needing to operate brokers, infrastructure, or core software services.