April 2025 Summaries

WarpStream, a pioneering data streaming system built on object storage without local disks, offers significant cost efficiencies compared to Apache Kafka, albeit with added latency. Initially, WarpStream's support for S3 Express One Zone (S3EOZ) reduced latency but also increased costs due to S3EOZ's pricing. Recent AWS price cuts for S3EOZ have made it more competitive, prompting WarpStream to revisit its benchmarks and total cost of ownership analysis. Results show WarpStream achieving significantly lower latency than other zero-disk systems, with a P99 Produce latency of 169ms. The system is cost-effective, with infrastructure costs totaling $2,961 per month, which is substantially cheaper than equivalent Kafka setups. WarpStream offers better durability and reliability, operating across multiple availability zones at no extra cost and auto-scaling to prevent disk space issues. Users can further reduce costs by opting for S3 standard, and detailed pricing comparisons are available via WarpStream's public pricing calculator.

Over the past decade, the author has tackled the challenge of efficiently removing logically deleted files from object storage in distributed systems like WarpStream, which is complicated by the necessity of maintaining compatibility with systems like Apache Kafka. Common methods such as using bucket policies or synchronous deletion have proven inadequate due to their limitations in handling variable retention policies and in-flight queries. Instead, more effective solutions include using a delayed queue, which allows files to be deleted from the metadata store and enqueued for physical deletion after a delay to accommodate live queries, and asynchronous reconciliation, which involves scanning the object store to identify and remove orphaned files. WarpStream initially relied on a reconciliation approach due to its scalability and ability to address orphaned files, but as customer demands increased, a hybrid method emerged, incorporating an "optimistic deletion queue" that reduces costs by preemptively deleting files after compactions. This approach, which balances efficiency, cost, and system integrity, has proven to be the author's preferred solution in managing object storage cleanup in distributed systems.

A memory leak in the WarpStream control plane, indicated by a linear increase in the HeapInUse metric, was investigated using pprof to compare heap profiles, revealing an unexpected retention of FileMetadata objects. These objects, linked with compaction jobs scheduled by the control plane, were found to be retained due to a goroutine leak in the deadscanner scheduler. This scheduler, responsible for removing untracked files from the object store, was inadvertently holding onto job references because it shared a job queue with the compaction scheduler. The issue arose when the deadscanner scheduler continued to run despite the job actor shutdown, as it was busy spinning in a loop due to backpressure from a full queue, leading to improper context cancellation handling. The solution involved modifying the job queue submission function to check for context cancellations before proceeding, which resolved the memory leak after deploying the patch. This case highlighted the importance of employing both broad and detailed diagnostic tools in debugging complex systems, as initial profiling did not reveal the issue until specific components were scrutinized.