Using New Relic Observability to reduce our Redis costs by 50%

New Relic's optimization journey involved addressing the challenges of running observability at scale, particularly focusing on their distributed tracing pipeline which processes massive streams of telemetry data. After implementing a new pipeline to simplify architecture and reduce operational complexity, they faced increased Redis storage and network costs. By using their own platform for monitoring, they embarked on a two-part optimization strategy: upgrading and right-sizing Redis clusters, and rethinking data storage by batching spans together to improve compression efficiency. The changes led to a 66 percent reduction in Redis memory and network usage, cutting annual costs by half without compromising system reliability or performance. This experience demonstrated the value of combining infrastructure improvements with smarter data storage practices, highlighting that significant efficiencies can be achieved through careful analysis and optimization of existing systems.