New Relic at re:Invent—our journey to AWS

In 2020, New Relic began transitioning its telemetry data platform to AWS to address scalability challenges due to rapid growth. At AWS re:Invent, Andrew Hartnett from New Relic detailed their adoption of a cell-based architecture to enhance scalability and geographical reach. The transition involved using Kafka to manage data pipelines and fully instrumenting systems to ensure observability, enabling teams to identify and address bottlenecks efficiently. Key strategies included planning for unexpected changes, starting with small, iterative steps, and maintaining open communication with customers. Additionally, New Relic focused on continuously building and decommissioning cells to improve platform reliability and reclaim engineering resources. Hartnett emphasized the importance of not only technical strategies but also effective communication during such large-scale migrations.