May 2024 Summaries

Zhibo Hu, an account executive at Confluent in Japan, has grown significantly in his role over the past year by engaging with customers and helping them transform their data architecture. He finds motivation in making businesses better through technology and enjoys working on a collaborative team where everyone contributes to common goals. In the future, Zhibo plans to become an industry expert in retail and telecommunications while continuing to provide value to customers throughout the sales process.

You're invited to a series of four meetups leading up to Current 2024, where you'll hear from speakers who will be presenting at the conference on topics related to their sessions. The meetups are held outside Confluent's office in Mountain View, California, with outdoor activities and summer-themed food and drinks, as well as a raffle for a complimentary ticket to Current 2024. The meetups cover different aspects of Apache Kafka, Apache Flink, and data streaming, including Kafka internals, the ecosystem, stream processing, and analytics. These events are an extension of Current 2024, offering bonus content and opportunities to network with speakers ahead of time.

The Data Streaming Awards recognizes organizations harnessing data streaming technology to drive business transformation and customer experience improvement. To nominate, submit an anonymous form by June 28, 2024, with evaluation criteria including innovation, impact, and interest. Winners will be announced at Current 2024 for categories such as Data Streaming Company of the Year and Best Individual Data Streaming System. Confluent has been named Microsoft's OSS on Azure Global Partner of the Year, reflecting its commitment to delivering open source-based solutions on Azure, amidst a shift where data streaming is becoming a critical foundation of modern business.

Effectively managing Confluent Cloud infrastructure presents challenges due to the complexity of deploying and maintaining components such as environments, clusters, topics, and authorizations. The Confluent Terraform Provider offers a streamlined solution for deploying Confluent Cloud resources by automating and organizing infrastructure management through best practices in Terraform and DevOps. Key strategies include segregating state files to prevent configuration errors, using modules for resource deployments, organizing configurations into environment-specific subdirectories, and employing dynamic resource creation with the for_each meta-argument. Additional practices involve leveraging lifecycle blocks for instance protection, avoiding hard-coded values by using variables and data sources, managing Kafka clusters within single Terraform workspaces for streamlined maintenance, and implementing robust secrets management strategies. Enforcing policy on resource configurations and maintaining consistent naming conventions further ensure security and compliance. Adopting these best practices leads to enhanced consistency, reduced manual errors, and improved security, ultimately optimizing Confluent Cloud infrastructure management.

The Defense Information Systems Agency (DISA) has launched Thunderdome, a comprehensive suite of IT and cyber-based technologies designed to provide unparalleled protection and reliability for the Department of Defense. This zero trust network access and application security architecture leverages enterprise Identity, Credential, and Access Management, commercial Secure Access Service Edge, and software-defined networking and security tools. The program aims to continually verify users to grant access to systems and data, utilizing real-time access to user and policy data. Confluent's data streaming platform will be a critical component of the Thunderdome architecture, enabling data integration, policy-based access control, and multi-tenant data management best practices. This initiative builds on DISA's previous zero trust cybersecurity reference architecture for the DoD, delivered in 2021, and marks an important step towards improving government cybersecurity.

Confluent Cloud's audit logs offer a crucial tool for proactively managing and troubleshooting incidents within large organizations by tracking user interactions and system changes. The blog post outlines how to set up a pipeline that transfers these audit logs from an external Confluent Cloud cluster to a personal one using Cluster Linking, then sends them to Splunk for alert configuration and management. By integrating with Splunk, users can create alerts for specific audit log events, such as cluster deletions or authentication failures, enabling immediate notification and rapid response to potential issues. The guide also provides technical details on configuring a Splunk Sink Connector, setting up alerts in Splunk, and offers workarounds for situations where standard practices cannot be followed. This approach aims to shift from reactive to proactive incident management, ensuring that suspicious activities are quickly identified and addressed in Confluent Cloud environments.

Apache Kafka has become the de facto standard for data streaming, but managing its capacity and scaling can be challenging due to unpredictable spikes in demand. Traditional approaches to capacity management, such as provisioning resources based on anticipated peak demand or scaling up and down during predictable periods, come with significant operational overhead and costs. In contrast, serverless clusters that autoscale to demand offer a more cost-effective solution, providing instant scalability without the need for manual intervention or upfront provisioning of resources. By automating capacity planning and scaling, these clusters can reduce infrastructure costs by 75%, engineering time and resources by over 50%, and downtime risk by up to 100x, resulting in significant cost savings that can be upwards of $750,000 per year. With the release of Enterprise clusters on both AWS and Azure, this solution is now more accessible and affordable than ever.

The new Confluent Cloud OpenSearch Sink Connector simplifies the integration between Confluent and OpenSearch, providing a seamless and fully managed way to stream data from Confluent Cloud to Amazon OpenSearch domains or other OpenSearch deployments. This connector reduces development cycles, operational burden, and cost-effectiveness, enabling customers to unlock powerful use cases such as real-time log analytics, security monitoring, social media analysis, and emerging areas like GenAI with RAG. With its flexibility, scalability, and support for various OpenSearch services, the connector represents a significant step forward in streamlining real-time data streaming to OpenSearch.

Confluent Cloud helped Moniepoint scale its architecture by providing a managed solution for processing transaction data in real-time. The company's initial infrastructure struggled with handling increasing transaction loads, leading to delays and manual effort to access and fix data. With Confluent, Moniepoint eliminated the need for read replicas, reduced replication delays, and improved scalability. The platform also enabled the company to consolidate disparate processes across its financial solutions without a large infrastructure team. By leveraging Confluent Cloud, Moniepoint was able to process millions of requests from its customers without downtime, positioning itself as a leader in the Nigerian ecosystem serving over 2 million businesses.

Confluent is a company that drives meaningful impact for its customers by bringing real-time data processing and streaming architectures to life, as told by Regional Sales Director Kimia Kia. Since joining Confluent, Kimia has expanded her technical expertise in this area and has been shaped by the company's culture, which she describes as all-hands-on-deck and supportive of continuous learning and growth. The company encourages diversity, equity, and inclusion through its global footprint and employee resource groups. Kimia is motivated to excel in her role by her team and the opportunity to work with customers who are passionate about building meaningful solutions with Kafka. She values the company's support for employee development and its emphasis on surrounding oneself with success.

The settlement process in capital markets is evolving towards a T+1 industry regulation, requiring real-time systems to manage increased trade volume and volatility. Confluent's data streaming platform enables financial institutions to create more effective settlement pipelines by providing a distributed asynchronous ecosystem, allowing for seamless integration and synchronization of disparate applications and systems. With the increasing pressure to settle trades faster, batch processing may not be able to meet future requirements, and Confluent's features such as Apache Kafka service powered by Kora provide 30x faster elasticity, 10x more reliability, and a 99.99% uptime SLA. The platform also enables building richer real-time data products, automating trade enrichment, and providing role-based access control to ensure the quality of the information received. By adopting Confluent's stream processing capabilities, financial institutions can unlock significant benefits around counterparty risk, improved liquidity, and lower capital requirements, ultimately driving an increased need for real-time systems and more efficient trade operations.

The integration of Amazon EventBridge with Confluent Cloud, part of the Connect with Confluent partner program, enhances the ecosystem by enabling seamless data streaming and real-time processing across various AWS services. EventBridge, a managed event routing service, allows customers to utilize Confluent’s tools for transforming and analyzing data, particularly for use cases involving AWS CloudTrail events. This integration supports the transmission of data via EventBridge API destinations to Confluent, leveraging Confluent's Produce REST API for real-time data streaming and analysis. Confluent Cloud's capabilities, such as Apache Flink, facilitate real-time anomaly detection and security monitoring, although the current integration is limited to REST and public Confluent Cloud clusters. Despite these limitations, the collaboration aims to strengthen security and compliance measures for organizations by enabling rapid detection and response to anomalies in vast data streams.

Ariel Gan, Regional Director at Confluent, has spent three years helping companies across industries solve technical and business challenges with data streaming. He emphasizes the importance of building relationships with customers to drive value within their teams and departments, which helps identify opportunities for leveraging data streaming and stream processing across organizations. Through his experience, Ariel has grown professionally, developed a passion for leadership, and credits Confluent's culture as unique in the sales space due to its collaborative network of peers and mentors. The company supports his ambitions through guidance from executive leaders and provides resources for employees to succeed, regardless of their background or prior experience.

The Apache Kafka community recently gathered in Bangalore, India, for the first-ever Kafka Summit in the country, with over 2,500 participants attending both in-person and online. Confluent's Chief Product Officer opened the summit, emphasizing the importance of data streaming and how it can transform businesses. The event featured a range of sessions on various topics, including real-time GenAI with Apache Kafka, Flink SQL, and Tableflow, which allows users to convert Apache Kafka topics to Apache Iceberg tables. Confluent Cloud for Apache Flink introduced AI Model Inference, enabling developers to simplify the development and deployment of AI/ML applications by providing a unified platform for both data processing and AI/ML tasks. The summit also highlighted the challenges of self-managing Flink and Kafka, which can be challenging due to the lack of immediate responsiveness from community support. Confluent Platform for Apache Flink was announced, offering long-term expert support and reducing operational complexity. Additionally, Confluent was named Microsoft's 2024 OSS on Azure Global Partner of the Year, recognizing its commitment to delivering outstanding open source-based applications and infrastructure solutions on Microsoft Azure.

Confluent Cloud has introduced Freight clusters, a new type of cluster designed for high-throughput, relaxed latency workloads that can save up to 90% on costs compared to self-managing Apache Kafka. These clusters utilize Confluent's Kora engine, which uses direct write mode to write data directly to object storage services like S3, bypassing local storage and avoiding replication between brokers. This approach eliminates inter-AZ replication traffic and reduces network costs, while introducing some latency trade-offs. Freight clusters are designed for high-volume use cases such as logging, telemetry, or feeding data lakes, and utilize Elastic CKUs to auto-scale capacity up or down with the workload, allowing customers to pay only for the resources they use when needed.

Deepthi Nidwannaya has been a significant contributor to Confluent's engineering team in India, driving the success of their billing capabilities and leading the re-architecting of the Commit Estimator platform. She prioritizes personal growth through her individual development plan and leverages Confluent's culture of empowerment and accountability to strive for excellence. With expertise in system development, deployment, and cloud solutions, Deepthi is eager to take on new responsibilities and drive strategic projects. Confluent's company culture values "Smart, Humble, and Empathetic" leadership, prioritizes employee growth, and encourages diversity, equity, and inclusion through policies such as comprehensive insurance coverage and mental health support. When interviewing candidates, she emphasizes the importance of evaluating not only technical skills but also interpersonal skills and cultural fit.