March 2022 Summaries

Seventh Sense, a predictive analytics platform for email marketers, transitioned to a streaming-first model to manage the increasing volume of data as the business grew from handling thousands to hundreds of millions of records. The company selected Redpanda as their streaming data platform due to its compatibility with Kafka APIs, lightweight nature, operational simplicity, and superior performance compared to alternatives like Kafka. This decision was driven by the need for a robust solution to efficiently process and analyze customer engagement data integrated from various CRM and marketing automation systems. Redpanda's capabilities allowed Seventh Sense to enhance developer productivity, reduce operational complexities, and explore new use cases by serving as an effective buffer for API data, optimizing performance across their query systems. The adoption of a streaming-first architecture, where the log serves as the primary data storage and databases act as materialized views, has proven effective in scaling and efficiently managing data, simplifying the development and operational processes for Seventh Sense.

Redpanda's introduction of tiered storage aims to efficiently unify historical and real-time data while reducing costs and improving reliability. By leveraging cloud object stores like Amazon S3 and Google Cloud Storage, Redpanda addresses data center reliability issues, enabling infinite data retention and decoupling storage capacity from the cluster's load capacity. This allows for more scalable and cost-effective infrastructure, as well as enhanced disaster recovery capabilities. The tiered storage architecture consists of components for both writing and reading data, such as the scheduler_service and remote_partition, which manage data uploads and retrievals from the cloud. The system uses metadata to ensure consistency and enables smart data retention by tracking uploaded segments. Future enhancements include full cluster recovery, faster data balancing, and the development of analytical clusters with read-only access to archived data, promoting elasticity and workload isolation. Overall, tiered storage offers developers and operators greater flexibility and efficiency in handling large volumes of data.