How LivePerson cut observability pipeline costs by benchmarking GCP infrastructure

LivePerson's observability team undertook a benchmarking study across five GCP machine types to optimize Logstash and Kafka performance, highlighting that infrastructure selection is crucial for cost optimization at scale. They discovered that the n4d-standard-2 (AMD Milan) machine type offered over 100% throughput improvement on Logstash compared to the e2-standard baseline, and similar gains were achieved on Kafka through the use of LZ4 compression instead of GZIP. These optimizations led to a significant reduction in processing costs, from $5.95 to $2.70 per 1,000 events per second, and allowed for a smaller Kafka cluster with reduced overhead. The study underscores the importance of recurring infrastructure benchmarking, as cloud providers frequently update instance families, and what was once cost-effective may no longer be competitive. This methodology is particularly relevant for high-volume observability workloads where compute efficiency directly influences cost and pipeline stability, and the insights gained are applicable beyond GCP to other cloud platforms like AWS and Azure.