Minimizing your cloud bill: how to calculate cloud data transfer costs

Cloud data transfer costs can be significant, especially when running distributed systems like Redpanda in a multi-AZ deployment. The main factor affecting these costs is whether the streaming platform should stay operational during an Availability Zone (AZ) outage. Using multi-AZ deployments increases resilience but incurs higher inter-AZ data transfer fees, whereas single-AZ deployments reduce costs but risk downtime during outages. Strategies to mitigate these expenses include optimizing data flow through efficient serialization formats like Apache Avro or Protocol Buffers, and enabling compression for data batches, which reduces the size of transmitted data. Additionally, Redpanda's follower fetching feature allows consumers to access data from the nearest replicas, lowering cross-AZ traffic. Leadership pinning, a more advanced technique, can be used to manage producer traffic by manually adjusting partition leaders within the same AZ. Together, these strategies can significantly reduce cloud costs, as demonstrated by a case where applying follower fetching and leadership pinning reduced monthly data transfer charges from $200 to $120.