February 2022 Summaries

Storyblocks, a media company founded in 2009, had to modernize its monolithic application by splitting it into microservices, which led to scaling challenges and technical debt. To address these issues, they adopted Apache Kafka and Confluent Cloud, which enabled them to reduce technical debt and improve data visibility. By using Kafka as an event-driven architecture, Storyblocks was able to create a single pipeline for all their data and communication between services, reducing complexity and improving scalability. With Confluent Cloud's managed service, they could focus on business-critical tasks rather than managing Kafka clusters. The company's experience highlights the importance of defining boundaries, refactoring applications, using Kafka as an event bus, and saving data to create a successful microservices architecture.

Monitoring Confluent Cloud Clusters is crucial for its performance and reliability. With the increasing popularity of Confluent Cloud, more Application Performance Monitoring (APM) solutions are being integrated with it to provide better insights into cluster behavior. This blog post discusses how to build a full monitoring solution using Confluent Cloud, Prometheus, Grafana, Datadog, Splunk, and New Relic. It provides step-by-step instructions on setting up each service, including creating credentials for the Metrics API, configuring Docker Compose, and deploying the services using docker-compose. The solution uses Prometheus to scrape metrics from the Confluent Cloud cluster and sends them to Grafana for visualization. Datadog and Splunk can also be integrated with Confluent Cloud to provide additional monitoring capabilities. New Relic's Open Telemetry Collector is another option for collecting Confluent Cloud metrics, offering a vendor-agnostic way to receive, process, and export telemetry data. The blog post provides detailed instructions on setting up each service and integrating them with Confluent Cloud, making it easier for organizations to monitor their clusters and gain insights into their performance and reliability.

Confluent is a fully managed Kafka service that Storyblocks chose over other hosted Kafka services due to its data governance capabilities, including Stream Quality, Stream Catalog, and Stream Lineage, which allow for better management of schema repositories and automated checks on downstream processes. Confluent also provides connectors to over 100 popular sources and sinks, reducing engineering time and technical debt. The company's Infinite Storage feature separates compute and storage in an event streaming platform, making storing increased amounts of data simpler and more cost-effective. Additionally, Confluent offers cloud-native features such as automated Kafka patching with zero downtime, capacity planning, and elastic scaling without the overhead of dealing with ZooKeeper, partitions, and JVMs. This expertise has been invaluable to Storyblocks, providing peace of mind and support for mission-critical use cases. With Confluent, Storyblocks can focus on innovating and pushing event-based streaming use cases forward.

The Salesforce data streaming pipeline described in the text utilizes a combination of Apache Kafka, Confluent Cloud connectors, ksqlDB, and BigQuery to process and load Salesforce data into a real-time data warehouse for analytics purposes. The pipeline uses change data capture (CDC) events from Salesforce to stream data into raw topics, which are then processed by ksqlDB applications to filter out gap events and reconcile records. The reconciliation stream is then combined with raw CDC data in BigQuery to form a complete snapshot of the Salesforce data. The pipeline also leverages Confluent Cloud connectors for Salesforce data ingestion, including CDC Source, Bulk API Source, PushTopic Source, Platform Event Source and Sink, SObjects Sink, and more. Additionally, the text discusses handling gap events by making API calls to Salesforce and using ksqlDB applications to process these events in real-time.