May 2024 Summaries

The author of the text discovered the Benthos project while browsing Reddit and was immediately intrigued by its lightweight stream processing framework. They Googled the project and fell in love with it, particularly its concise and entertaining documentation. The author met Ashley Jeffs, the primary author, at Kafka Summit London and decided to sponsor the project to integrate it into their own product, WarpStream. However, after Redpanda acquired Benthos, the author was surprised by the swift changes made to the project, including commercially licensing some integrations and rebranding the Discord community. The author chose to fork the project, maintaining it as a free, open-source MIT-licensed repository called Bento3, which will continue to carry on where Benthos left off.

Tiered storage solutions won't fix Kafka's issues, and instead, introduce new challenges such as increased unpredictability and difficulty in management. Zero Disk Architectures (ZDAs) offer a better solution by eliminating disks altogether, achieving zero disks with everything running directly through object storage. ZDAs enable auto-scaling, radically flexible topologies, and deployment models, making it easier to manage data streaming workloads. They also facilitate HTAP capabilities, enabling transactional and analytical workloads to operate on the same dataset without replication delay. With ZDAs, developers can create single logical clusters that span traditional cloud networking boundaries in a cost-effective manner. Additionally, ZDAs enable feature-rich agents that can handle writes or reads for any topic-partition, making it feasible to scale out writes or reads on a moment's notice. They also bring security and compliance benefits by isolating producers from consumers and allowing customers to manage their own compute/data plane with zero cross-account IAM access or privileges granted to the vendor. Overall, Zero Disk Architectures are transforming the data streaming space by flipping pricing, capabilities, and deployment models entirely on their head.

Pixel Federation, a developer of popular mobile games, initially used Apache Kafka as their message bus for real-time data streaming infrastructure, leveraging AWS's managed Kafka offering, MSK. As their usage grew, so did the complexity and costs associated with managing this setup, including high costs due to scaling requirements and inter-VPC networking fees. After migrating to WarpStream, they implemented an Agent Groups architecture that uses a shared object storage bucket as both storage and networking layer, resulting in significant cost savings of over 6x compared to their previous MSK solution, while also offering additional benefits like seamless auto-scaling and no manual partition rebalancing. This migration allowed Pixel Federation to flex their cluster across multiple VPCs and AWS accounts with ease, making it a more efficient and cost-effective solution for their data infrastructure needs.

WarpStream has introduced WarpStream Managed Data Pipelines, a fully-managed SaaS experience for Bento, which is a lightweight stream processing framework offering much of the functionality of Kafka Connect and additional stream processing capabilities like single message transforms, aggregations, and more. With this feature, WarpStream users can now directly connect with external systems, perform data transformation, and stream data into downstream systems using a simple YAML configuration. Managed Data Pipelines provide a cloud-native, BYOC-managed version of Bento, enhanced with WarpStream's signature data plane / control plane split, allowing pipeline configuration, version control, clustering, and deployment to be administered remotely using a SaaS UX while pipelines run in the user's cloud account. This feature is available for free, with no credit card required, and can be started in just a few minutes.