August 2024 Summaries

Busie, a New York-based company, has developed an omnichannel platform for ground transportation providers using data streaming with Confluent Cloud. The Charter Platform allows providers to offer end-to-end online booking processes, streamlining operations and improving customer experience. By leveraging data streaming, Busie can provide features such as dynamic pricing, real-time notifications, and analytics dashboards. With the support of Confluent's Data Streaming Platform, Busie is able to develop their product at a rapid pace while managing resources efficiently.

Apna, India's largest jobs site, has successfully re-architected its platform using technologies like Apache Kafka and a universal data lakehouse for faster growth and innovation. The company moved from a monolithic software architecture with batch updates to microservices, streaming data pipelines, and a data lakehouse architecture for data integration. This change allowed Apna to outperform competitors by improving customer experience while reducing costs and time-to-market for new business solutions. The organization's new data infrastructure is built on Confluent Cloud and Onehouse's universal data lakehouse, both of which are open source technologies that support key use cases like change data capture (CDC) for operational databases. Apna's adoption of these managed services has enabled the company to maintain high levels of engagement with its partners while focusing on industry-specific innovation and business needs.

The blog discusses the integration of Amazon DynamoDB with Confluent's data streaming platform to build event-driven architectures for real-time insights and decision-making. It explores a fourth option for capturing and transferring data changes from DynamoDB to Confluent: Confluent Cloud’s source connector for Amazon DynamoDB Change Data Capture (CDC). This fully managed service simplifies the process of capturing and transferring data changes, reduces operational complexity, and provides seamless integration. The source connector supports no-code integration with Confluent, cost-effective streaming with multi-table support, multiple usage modes, enhanced developer productivity, custom offset support, IAM AssumeRole support, and more. An example use case of the connector is provided along with steps to set up the source connector in a Confluent Cloud environment. The integration offers various advantages such as hydrating data lakes, facilitating real-time data processing, expanding the range of data sources, and broadening potential destinations for data. This feature is available across all major clouds.

The text discusses various topics related to data streaming and Apache Kafka. It mentions the evolution of change data capture (CDC) methods, the introduction of first-class data products for resilient pipelines, and how innovations in Kora enable new Confluent Cloud Freight clusters that can save up to 90% at GBps+ scale. The text also highlights the process of writing Kafka Improvement Proposals (KIPs) and integrating Apache Kafka with other platforms like AWS DynamoDB, Spring Boot with Kotlin, and Confluent Cloud. Additionally, it talks about the revamping of Skai's interactive ad-campaign dashboard using Apache Kafka and an in-memory database, as well as the new features and improvements in Apache Flink 1.20 and Apache Kafka 3.8.0 releases. The text also discusses the general availability of Confluent Platform 7.7 and its latest key features, including enhanced security with OAuth support and a new Connector. Lastly, it mentions the upcoming Q2 2024 Confluent Cloud launch and how to use OpenSearch Ingestion to integrate Confluent with Amazon OpenSearch.

In a webinar titled "How to build RAG-enabled GenAI with Confluent, Flink & MongoDB," Janis Hom shares insights on her growth since joining Confluent. She highlights the benefits of remote work and the opportunity to integrate governance, risk, and compliance (GRC) into overall trust and security at the company. Hom also discusses the unique culture at Confluent that encourages open disagreements and fosters diversity, equity, and inclusion through various internal events. Additionally, she shares her motivation for excelling in her role, which stems from working with highly motivated colleagues and ensuring quality work.

The Spring Framework and Apache Kafka have been integrated for years, allowing organizations to transition from traditional microservices to event-driven architectures with minimal disruption. A new guide is available on how to build GenAI apps faster using Spring Boot, Apache Kafka, Confluent Cloud, and Kotlin, leveraging the expressive nature of the Kotlin language and coroutines to simplify development. The guide covers producing events to Apache Kafka topics through Confluent Cloud, registering schemas with Schema Registry, and consuming events using the `@KafkaListener` annotation. It also discusses how to configure Spring Boot applications for Avro-serialized events, including generating Java classes from Avro schema documents and configuring producer and consumer settings. The guide provides a simple example of sending events to Kafka topics from a Spring application and demonstrates how to use Spring Cloud Config to import external configuration as environment variables.

Confluent Cloud supports over 80 pre-built, fully managed connectors for data integration with Apache Kafka, which are built on the open source Kafka Connect framework and thrive with an active community. The Amazon S3, Snowflake, Google Cloud Storage, and Google BigQuery sink connectors consistently rank near the top of the list, while organizations often use multiple connectors to build streaming data pipelines. Leveraging fully managed connectors is the fastest and most efficient way to integrate data systems and apps with Kafka by eliminating operational burdens. Confluent's Stream Governance enables organizations to enforce high data quality and stream processing with Apache Flink, while custom connectors can be used for specific use cases. The Datagen Source Connector is a great place to start for developers deploying their first connector on Confluent Cloud, and the company has streamlined the connector configuration workflow to reduce friction and save time. Beyond cloud connectors, organizations may require self-managing certain connectors in their own environment due to security requirements, and Confluent Hub offers hundreds of connector plugins where users can filter by licensing type and check their deployment options.

This week in Las Vegas, Google is hosting their annual Google Cloud Next ʼ24 conference, bringing together leaders from across the IT industry to learn about and explore the newest innovations in and around the Google Cloud ecosystem. Confluent has been named a Leader in two different IDC MarketScape reports for its data streaming platforms, and will be attending Current 2024 in Austin, Texas, where it will bring the community together to discuss Apache Kafka and Apache Flink. Confluent is also thrilled to have been named Microsoft’s 2024 OSS on Azure Global Partner of the Year, recognizing its commitment to delivering open source-based applications and infrastructure solutions on Microsoft Azure. The company has collaborated with Amazon Web Services (AWS) to make cloud services and data management easier for government agencies. Additionally, Confluent is hosting Kafka Summit in Bangalore, India, which drew over 23,580 participants, both in-person and online, and features the company's engineering team working on its data streaming platform's billing capabilities.

The Current 2024 webinar will take place on September 17-18 in Austin, Texas, and features a packed two-day agenda with nearly 100 sessions covering topics from message broker basics to implementing real-time systems with Kafka and Flink. The event includes morning keynotes on data infrastructure and the developer's role in data streaming, as well as community meetup hub, training and certifications, birds-of-a-feather lunches, a sundowner party, diversity & inclusion programming, and a closing session raffle. Registration is now open with promo code DEVREL offering $450 off registration prices. The event aims to bring together tech leaders, industry giants, and startups to discuss real-time event streaming and event-driven architectures.

We managed an online advertising campaign using an interactive tool that let us pivot, analyze, filter, and display performance across all channels. However, as we added features and accrued more data, our UI updates became slower due to the relational databases' inability to scale. We investigated change-data-capture replication with fast in-memory databases using Apache Kafka and Debezium connector. Our initial architecture improved performance but had limitations, including high maintenance costs and inefficient utilization. To address these challenges, we consolidated tables into shared topics and pipelines, reducing the number of topics from 90K to 1.8K. We also implemented a new topic type for heavy-load tables with six partitions and a caching mechanism to reduce API calls. Our re-architecture reduced topics by over 90% without sacrificing performance, freeing up brokers and reducing maintenance time and cloud budget.

The Confluent team has redesigned the Confluent Cloud Homepage to improve user experience, making it more intuitive and user-friendly, with features such as a clear navigation, relevant information, improved wayfinding, and enhanced discovery capabilities. The new design focuses on empowering users to be self-sufficient by providing workflows that support their expertise and allow them to extract maximum value from their data with minimal friction. The redesign includes various enhancements, including improved health indicators, favorites and recently visited resources tiles, a Discover data tile, Data access requests tile, Recommended actions tile, and Blogs tile, which aim to streamline navigation, improve engagement, and keep users informed about recent product launches and updates.

Confluent Cloud and BigQuery are revolutionizing data handling by offering a powerful solution for real-time data ingestion, processing, and analysis through continuous queries. This integration enables organizations to overcome challenges in moving data from traditional data warehouses to real-time processing platforms, resulting in faster insights and decision-making. By combining Confluent Cloud with BigQuery's continuous query feature, businesses can unlock new business opportunities, streamline data pipelines, and improve application performance, ultimately gaining a competitive edge with real-time, data-driven decision-making.

The latest release of Apache Flink 1.20 has brought several significant improvements to its usability, performance, and feature set, setting the stage for the upcoming major release of Flink 2.0. The update introduces a unified way to specify bucketing strategies for tables, making it easier to manage data distribution and freshness requirements. Additionally, Flink now supports computing aggregations on non-keyed streams in parallel and offers improved state management with multiple configurations to control upload behavior. The community has also fixed several issues, including a longstanding problem with RocksDB state backend, and made final preparations for the upcoming Flink 2.0 release. Furthermore, Confluent Cloud for Apache Flink supports AI model inference and enables the use of models as resources in Flink SQL, expanding its capabilities for machine learning and data processing.

The Apache Flink community has released version 1.20, focusing on usability enhancements for developers and operators, ahead of the anticipated Flink 2.0. The update features notable improvements in Flink SQL, including a new bucketing strategy (FLIP-376) and the ability to specify data freshness requirements (FLIP-435) through materialized tables. Additionally, FLIP-380 enables parallel computation of aggregations on non-keyed streams in batch mode, addressing the deprecation of the DataSet API. Other significant changes include configurations to manage state file uploads for checkpoints (FLIP-306) to reduce load on storage systems, and enhancements in recovery processes from JobMaster failures (FLIP-383). The update concludes the community's efforts to deprecate outdated APIs and refine configurations, as preparations for Flink 2.0 continue. The release is supported by contributions from numerous developers and further integrates with Confluent Cloud for AI model inference in Flink SQL.