How LivePerson cut observability pipeline costs by benchmarking GCP infrastructure

LivePerson's observability team undertook a benchmarking study across five GCP machine types to optimize Logstash and Kafka performance, highlighting that infrastructure selection is crucial for cost optimization at scale. They discovered that the n4d-standard-2 (AMD Milan) machine type offered over 100% throughput improvement on Logstash compared to the e2-standard baseline, and similar gains were achieved on Kafka through the use of LZ4 compression instead of GZIP. These optimizations led to a significant reduction in processing costs, from $5.95 to $2.70 per 1,000 events per second, and allowed for a smaller Kafka cluster with reduced overhead. The study underscores the importance of recurring infrastructure benchmarking, as cloud providers frequently update instance families, and what was once cost-effective may no longer be competitive. This methodology is particularly relevant for high-volume observability workloads where compute efficiency directly influences cost and pipeline stability, and the insights gained are applicable beyond GCP to other cloud platforms like AWS and Azure.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.