Transitioning to the cloud: New Relic's journey to AWS

Wendy Shepperd, GVP of Engineering at New Relic, discusses the company's transition to Amazon Web Services (AWS) and the adoption of cell architecture to address scalability challenges. As New Relic's platform, NRDB, handles over 3 billion data points per minute and 150 petabytes of data monthly, the move was driven by the need to overcome limitations of a single Kafka cluster in their data center. The transition involved using AWS-managed services like Amazon Elastic Container Service for Kubernetes (EKS) and Amazon Managed Streaming for Apache Kafka (MSK), allowing New Relic to create multiple instances of NRDB across regions. Observability through New Relic's platform facilitated the migration by enabling performance comparisons and quick issue remediation, ultimately proving successful during unexpected traffic spikes. The transition highlights the importance of cloud cost management and iterative planning, with advice for other practitioners to focus on application instrumentation, setting service level objectives, and optimizing cloud costs. Additionally, best practices for cell architecture are shared, emphasizing the need for regular updates to maintain security and efficiency.