April 2024 Summaries

Tiered storage in Apache Kafka aims to reduce costs by moving historical data to object storage, but it often ends up being penny-wise and pound-foolish. It can make Kafka more difficult and complicated to deal with, rather than simpler, and introduces new operational problems and failure modes. While it may reduce some costs, such as cloud networking fees, these savings are often offset by increased complexity and unpredictability. In practice, tiered storage does not live up to its promises of simplifying operations or making Kafka more efficient, and many users have been disappointed with the results. Instead, some vendors are rebranding existing products as "tiered storage" without actually implementing any significant changes.

EBS volumes are expensive due to their durability features, which provide value through detachability and resizing, but at a cost of $0.08/GiB or $0.24/GiB when considering replication. The high cost is because EBS was designed for traditional datacenter environments, not cloud-native ones. Object storage like S3 in AWS can be 24 times cheaper, making it more cost-effective for large data volumes. Shedding local disks entirely may be necessary for optimal economics in the cloud, but "tiered storage" approaches often lead to operational headaches and are an architectural dead-end.

This blog presents a real-time security threat monitoring system that integrates RisingWave, WarpStream, and Grafana, providing scalable and responsive capabilities for quick identification and mitigation of security risks in web applications. The system leverages the synergies of these tools to offer comprehensive security monitoring capabilities through unified dashboards and real-time analytics, enabling enhanced threat detection and response.