February 2022 Summaries

An account of an event-driven gym-booking platform illustrates how real-time updates could give customers, instructors, receptionists, and sales teams immediate, tailored visibility into bookings, attendance, and business performance, although the project ultimately failed largely for management reasons. The author argues that while relational databases were capable and valuable, using them for event streaming required custom solutions for polling new events, maintaining processing offsets, calculating current-state aggregates, recovering in-memory state after outages, and scaling notification delivery. Apache Kafka and ksqlDB are presented as alternatives designed around these needs, offering consumer offsets, continuously updated SQL-based aggregates, persistent state recovery, partitioning, consumer groups, replication, and exactly-once processing semantics. The central point is that selecting purpose-built streaming infrastructure can reduce the engineering effort needed to operate a real-time system, allowing teams to concentrate on their distinctive product and user experience rather than rebuilding distributed data-processing capabilities.

Storyblocks, a media company founded in 2009, had to modernize its monolithic application by splitting it into microservices, which led to scaling challenges and technical debt. To address these issues, they adopted Apache Kafka and Confluent Cloud, which enabled them to reduce technical debt and improve data visibility. By using Kafka as an event-driven architecture, Storyblocks was able to create a single pipeline for all their data and communication between services, reducing complexity and improving scalability. With Confluent Cloud's managed service, they could focus on business-critical tasks rather than managing Kafka clusters. The company's experience highlights the importance of defining boundaries, refactoring applications, using Kafka as an event bus, and saving data to create a successful microservices architecture.

Monitoring Confluent Cloud Clusters is crucial for its performance and reliability. With the increasing popularity of Confluent Cloud, more Application Performance Monitoring (APM) solutions are being integrated with it to provide better insights into cluster behavior. This blog post discusses how to build a full monitoring solution using Confluent Cloud, Prometheus, Grafana, Datadog, Splunk, and New Relic. It provides step-by-step instructions on setting up each service, including creating credentials for the Metrics API, configuring Docker Compose, and deploying the services using docker-compose. The solution uses Prometheus to scrape metrics from the Confluent Cloud cluster and sends them to Grafana for visualization. Datadog and Splunk can also be integrated with Confluent Cloud to provide additional monitoring capabilities. New Relic's Open Telemetry Collector is another option for collecting Confluent Cloud metrics, offering a vendor-agnostic way to receive, process, and export telemetry data. The blog post provides detailed instructions on setting up each service and integrating them with Confluent Cloud, making it easier for organizations to monitor their clusters and gain insights into their performance and reliability.

Confluent is a fully managed Kafka service that Storyblocks chose over other hosted Kafka services due to its data governance capabilities, including Stream Quality, Stream Catalog, and Stream Lineage, which allow for better management of schema repositories and automated checks on downstream processes. Confluent also provides connectors to over 100 popular sources and sinks, reducing engineering time and technical debt. The company's Infinite Storage feature separates compute and storage in an event streaming platform, making storing increased amounts of data simpler and more cost-effective. Additionally, Confluent offers cloud-native features such as automated Kafka patching with zero downtime, capacity planning, and elastic scaling without the overhead of dealing with ZooKeeper, partitions, and JVMs. This expertise has been invaluable to Storyblocks, providing peace of mind and support for mission-critical use cases. With Confluent, Storyblocks can focus on innovating and pushing event-based streaming use cases forward.

Data architectures often become fragmented as organizations incrementally add applications, data stores, and environments connected through point-to-point integrations, creating silos that can limit innovation, rely on stale batch data, and prevent real-time application capabilities. The passage compares this condition to disk fragmentation and argues that organizations can “defragment” their architecture by adopting a unified, real-time data plane that makes data accessible across systems while preserving existing operations. It emphasizes selecting a platform with broad pre-built connectors, support for on-premises and cloud deployments, consistent APIs, data mobility, and high streaming performance to avoid costly custom integrations and enable future flexibility. Migration can occur incrementally with no downtime by onboarding workloads through tools such as Kafka Connect and applying stream processing through ksqlDB, potentially increasing the platform’s value as more teams and data sources participate. Confluent presents its cloud-native data streaming platform as a solution for hybrid and multicloud modernization, claiming benefits including lower costs, reduced risk, greater revenue opportunities, and improved customer experiences.

The Salesforce data streaming pipeline described in the text utilizes a combination of Apache Kafka, Confluent Cloud connectors, ksqlDB, and BigQuery to process and load Salesforce data into a real-time data warehouse for analytics purposes. The pipeline uses change data capture (CDC) events from Salesforce to stream data into raw topics, which are then processed by ksqlDB applications to filter out gap events and reconcile records. The reconciliation stream is then combined with raw CDC data in BigQuery to form a complete snapshot of the Salesforce data. The pipeline also leverages Confluent Cloud connectors for Salesforce data ingestion, including CDC Source, Bulk API Source, PushTopic Source, Platform Event Source and Sink, SObjects Sink, and more. Additionally, the text discusses handling gap events by making API calls to Salesforce and using ksqlDB applications to process these events in real-time.

Thierry Mbemba describes his progression from an inbound sales development representative at Confluent in 2018 to an enterprise account executive responsible for winning and expanding relationships with large customers. His career developed through taking on both inbound and outbound sales work, briefly leading a team, and eventually managing full sales cycles for increasingly large companies, with three promotions in four years. Mbemba attributes Confluent’s opportunity to the growing importance of real-time data management for businesses and notes that the company expanded from roughly 270 employees to about 2,000 during his tenure. He characterizes sales as demanding because of performance targets and uncertainty, citing a period with no closed deals during the pandemic before he secured one of the company’s largest transactions. He emphasizes perseverance, collaboration, internal champions, coaching, customer engagement, and using specialized colleagues and sales tools to address customer needs, illustrating these principles with a year-long pursuit of a prospect that ultimately became a major deal.