October 2024 Summaries

Confluent Platform is a complete data streaming platform that complements Apache Kafka with advanced features such as automatic observer promotion (AOP) and cluster linking, enabling sophisticated multi-data center architectures. These architectures allow service providers to offer more profitable fully managed Kafka services by delivering higher service level agreements (SLAs) in terms of recovery time objectives (RTOs), recovery point objectives (RPOs), and availability. Confluent Platform empowers service providers to reduce operating costs and achieve operational excellence through multi-tenancy, deployment and operation automation, flexible scaling, deployment on cost-effective ARM infrastructure, self-balancing capabilities, tiered storage, comprehensive security capabilities, and seamless data integration with pre-built connectors. By leveraging Confluent, service providers can deliver modern, scalable, and efficient data solutions that meet the diverse needs of their customers, drive greater business value for their clients, and provide exceptional services to their customers. The Confluent OEM Program empowers MSPs and CSPs to quickly unlock new revenue streams by integrating the industry's leading data streaming platform into their customer offerings, with minimal investment and risk.

As a product manager on the Confluent Platform team, Surabhi Singh collaborates closely with cross-functional teams to build solutions that enable customers to monitor, diagnose, and optimize their infrastructure for smooth, uninterrupted streaming operations. Success is defined by customer adoption of new features, improvements in platform stability and performance, and positive feedback. Since joining Confluent, Surabhi has deepened her understanding of data streaming and open source technologies, learning about Apache Kafka and Apache Flink, and exploring emerging use cases from customers. She recommends a supportive professional environment to thrive, with collaboration and accountability being critical in remote-first settings. Confluent fosters diversity, equity, and inclusion through intentional hiring, inclusive leadership, and company-wide incentives, empowering employees to contribute authentically. Surabhi values the company's growth opportunities and resources for learning and development, which motivate her to excel in her role.

The headless data architecture is a formalization of a data access layer at the center of an organization, providing consistent data access for operational and analytical use cases. It requires identifying work to shift left from downstream, creating a single standardized set of data through streams and tables. This approach reduces downstream costs by shifting work to the left, simplifying data creation, access, and use. The traditional multi-hop architecture is slower, expensive, brittle, and can lead to similar-yet-different data sets, which can cause conflicts and loss of trust. A shift-left approach provides a more cost-effective way to create, access, and use data, with the logical top level being the data product, composed of a stream and its related table. Data products are trustworthy, standardized, and reusable, simplifying getting the data needed by teams and services. Shifting left is modular and incremental, allowing selective choice of loads to shift left, validation, and replacement of existing jobs. This approach unlocks unparalleled data access across the organization, providing freedom from coordinating multiple copies of data and using suitable processing or query engines.

AWS GameDay is an immersive experience where cloud experts test their knowledge in real-world scenarios using Amazon Web Services (AWS) and Confluent Cloud. It's a team-based, interactive exercise designed for technical professionals to explore AWS services, architecture patterns, and best practices in a gamified, risk-free environment. Participants gain practical experience with AWS services, build problem-solving skills, and collaborate effectively as they tackle real-world challenges. The event is open-ended and non-prescriptive, providing freedom to explore and think creatively. Through an interactive approach, attendees deepen their understanding of AWS and partner products and services, enabling them to build effective cloud solutions. The event offers numerous benefits, including risk-free experimentation, innovation, and the chance to interact with Confluent features such as Schema Registry, Flink, and fully managed connectors.

The headless data architecture is an emerging concept that separates data storage, management, optimization, and access from the services that process and query it. This allows for a single logical location to manage permissions, schema evolution, and table optimizations, making regulatory compliance simpler. A headless data architecture can encompass multiple data formats, including streams and tables, providing flexibility in choosing the format suitable for operational, analytical, or hybrid use cases. Apache Kafka is an open-source distributed event-driven streaming platform that has a headless data model since its inception, allowing producers to write about topics independently of consumers. The producer acts as a fully independent head, while consumers are also independent. To support full streaming capabilities, events require well-defined schemas and metadata catalogs. Tables can be integrated into the headless data architecture using Apache Iceberg, which provides table storage and optimization, catalog management, transactions, time travel capabilities, and pluggable data layers. The main benefits of a headless data architecture include saving money and time by not copying data around, eliminating similar-yet-different datasets, and choosing the most suitable processing engine for each use case. A headless data architecture differs from a data lake architecture in that any service can use the data, tables and streams are used interchangeably, and the data layer is modular and composed of different data sources. This architecture enables building data lakes and warehouses by plugging in Iceberg tables, allowing businesses to invest in their own headless data architectures and providing modularity, reusability, structure, and easy access to both streams and tables.

The Kafka consumer plays a crucial role in reading data from Kafka topics, but it's not as simple as just setting up a client and calling `consumer.poll()`. The consumer must first determine which topics and partitions to consume from, and this is where the configuration parameters `group.id` and `partition.assignment.strategy` come into play. These parameters control how partitions are assigned to consumers within a group, and there are several strategies available, including RangeAssignor, RoundRobinAssignor, StickyAssignor, CooperativeStickyAssignor, among others. Additionally, consumers must issue fetch requests to get the relevant offsets from the internal `__consumer_offsets` topic, and they can configure parameters such as `auto.offset.reset` to determine where in the Kafka topic to start reading from. Once these settings are determined, consumers send fetch requests using binary protocol over TCP, which result in a request-response process with various configuration parameters controlling the amount of data returned, such as `fetch.min.bytes`, `fetch.max.bytes`, and `max.partition.fetch.bytes`. The consumer must also monitor metrics related to offsets, partition assignment, and request handling to ensure optimal performance.

The latest development in Apache Kafka's migration to self-managed metadata in KRaft is now production-ready. The new consensus protocol replaces ZooKeeper for metadata management and cluster consensus, allowing for improved scalability, stability, and security. Confluent has successfully migrated thousands of clusters to KRaft without downtime, demonstrating its readiness for production use. With KRaft, Kafka can handle millions of partitions across a fleet of clusters with increased efficiency, reducing complexity and improving overall system stability. The migration process is now available through various tools, including Confluent's CFK and Ansible Playbooks, making it straightforward for experienced operators to transition to KRaft.

The text discusses various strategies for handling bad data in event streams, which are different from batch processing due to their immutable nature. The first technique is prevention, which involves using schemas, tests, and data quality constraints to ensure well-defined data from the start. This approach saves headaches and problems in batch processing. The next layer of defense is event design, particularly state events, which prevent bad data by designing events that produce well-defined data in the first place. State events also enable event-carried state transfer and allow consumers to infer deltas from previous events. Compaction, a process in Apache Kafka, can be used to delete older versions of data with the same record key. However, fixing delta-style events is more challenging due to their tight coupling with business logic and the inability to compact them. Two strategies for repairing bad delta events are building-forward techniques or rewinding, rebuilding, and retrying the topic. The latter approach requires significant intervention and can be expensive and complex. Effective event design and prevention remain crucial in dealing with bad data in event streams.

Seema Acharya, software engineering manager at Mercedes-Benz R&D North America, discussed how Mercedes-Benz uses a data streaming platform to collect and process data from various sources, creating personalized experiences for customers. The platform handles 800 terabytes of data per month and provides real-time insights into customer behavior, enabling the company to drastically reduce time-to-market for new projects. Industry experts shared examples of how data streaming platforms can help businesses deliver personalized entertainment experiences, manage market data, detect fraud in real-time, and extract value from data at scale. The industry is shifting towards a "shift-left" strategy, where data processing and governance are moved closer to data generation time to eliminate duplicate pipelines, reduce risk, and leverage high-quality data products. Companies like Confluent are making it easier for businesses to harness the power of their data by providing simple and powerful data streaming solutions. The acquisition of WarpStream has expanded Confluent's offerings to include a data streaming solution for every company, regardless of their needs or infrastructure.

This webinar focuses on protecting sensitive data using CSFLE (Confluent Schema for Format and Language Enforcement) with Spring Framework, Confluent Cloud, and the Kotlin language. It explores how to create Kafka Streams topologies that persist results in a state store, allowing for querying of counts for specific words. The presentation also delves into joining streams of typed data using Avro schemas, demonstrating how to track check-in events for high-level members with enriched check-in events. Throughout the webinar, Kotlin's concise syntax and null safety features are highlighted as advantages over Java in building APIs and topologies. The demonstration showcases how to write unit tests for Kafka Streams topologies, including testing for known and unknown member cases, using JUnit 5's @ParameterizedTest annotation.

Confluent's latest enhancement integrates AWS IAM roles with its managed connectors, allowing for temporary security credentials to be used, reducing the risk of long-term credential exposure and operational burden of key management. This feature enables users to adopt a more secure approach to managing access to their data flows between AWS and Confluent Cloud, by using IAM roles to delegate access and limit the use of long-term credentials. The integration also reduces the attack vector by eliminating the risk of leaking access keys, provides enhanced security with limited lifetime for IAM roles, and simplifies access management for users. To set up this feature, users need to create an IAM role in AWS, register it within Confluent Cloud, generate an integration ID, and update the trust policy with Confluent IAM role configurations. The integration allows customers to securely source from or sink to targets within their cloud environments, eliminating credential leakage risks and reducing operational overhead.

Bad data can have serious consequences in event streams, leading to incorrect decisions and significant negative impacts on business operations. To prevent bad data from entering an event stream, schemas are essential, along with testing, validation rules, and data contracts. These strategies significantly reduce error incident rates by preventing producers from writing bad data and making it easier for consumers to focus on using the data. By adopting these prevention techniques, businesses can save time and effort in dealing with bad data, and instead focus on getting actual work done with their data.

The Apache Flink Table API is a unified relational API for both stream and batch processing, offering a higher-level abstraction compared to the DataStream API. It allows developers to express complex data processing logic in a declarative manner using Java or Python, making it particularly appealing for those who want to efficiently manipulate data without getting bogged down in intricate implementation details. The Table API is closely integrated with Flink SQL and the DataStream API, enabling users to leverage the strengths of each API as needed. It provides essential features such as windowing capabilities, set operations, and user-defined functions, making it suitable for handling both continuous streams of data and static datasets efficiently. The Table API also supports asynchronous operations, event time processing, and sophisticated windowing operations, making it ideal for scenarios like real-time analytics or continuous ETL. With its unified relational interface, developers can simplify complex data operations and build scalable, flexible, and efficient applications.