June 2017 Summaries

The concept of a summer reading list has evolved to include professional development and technical skills, allowing individuals to boost their knowledge and make themselves more marketable in the future. The New Relic summer reading list features a diverse range of books covering topics such as software development, business strategy, and self-improvement, including "Working Effectively With Legacy Code", "Good Strategy, Bad Strategy", and "Machine, Platform, Crowd". These books offer insights into topics like functional programming, microservices design, and the effects of automation on the economy. The list also includes some "traditional" light summer reading options, such as "Gödel, Escher, Bach: An Eternal Golden Braid" and "Stop Chasing Carrots: Healing Self-Help Deceptions With a Scientific Philosophy of Life". Individuals are encouraged to share their own book recommendations using the hashtag #RelicReadingList.

Summer reading lists, once a staple of grade school, remain popular for personal and professional development, offering opportunities to enhance skills and knowledge. The Relic Reading List features a selection of books recommended by New Relic employees, covering technical topics like legacy code, microservices, and functional programming, as well as business-oriented books on strategy and team dynamics. Titles such as "Good Strategy, Bad Strategy" and "Peopleware" offer insights into effective business practices, while "Machine, Platform, Crowd" and "Scale" explore digital transformation and scaling in various contexts. For lighter reading, "Stop Chasing Carrots" addresses self-help and fulfillment, and "Gödel, Escher, Bach" delves into the interplay of math, cognition, and biology. These recommendations aim to inspire and equip readers with valuable insights for their careers and personal growth.

New Relic's Dynamic Baseline Alerts are now generally available, allowing customers to set dynamic alert thresholds based on predicted application behavior using historical data. This feature reduces guesswork and alert fatigue by providing real-time feedback on threshold settings, enabling users to create alerts with confidence and establish dynamic thresholds that account for seasonal changes and long-term trends in application behavior. With minute-scale granularity, Dynamic Baseline Alerts can learn short-term patterns and seasonalities hiding in metric data, making it easier for busy APM users to manage their applications without having to guess at establishing suitable alerting thresholds.

New Relic has introduced Dynamic Baseline Alerts, a feature designed to enhance monitoring by setting dynamic alert thresholds based on historical application data. Unlike static thresholds, which may not account for seasonal changes or trends in application behavior, Dynamic Baseline Alerts use predictive modeling to establish baselines, allowing operations teams to optimize alert conditions without the guesswork. This capability enables users to visually adjust thresholds with real-time feedback, reducing alert fatigue and improving monitoring efficiency. The feature, now generally available for New Relic PRO subscribers, provides minute-scale granularity to capture short-lived anomalies and enhances accuracy over time as more data is collected. New Relic emphasizes that this innovation simplifies the process of establishing effective alerts, reflecting a commitment to leveraging artificial intelligence and machine learning for improving application performance monitoring.

New Relic has improved its Dynamic Baseline Alerts system by implementing auto-discovered seasonality and an ensemble algorithm that chooses the best fit for each time series data stream every minute. This allows for more accurate predictions and better performance, while also making it easier for customers to use the service without having to manually select algorithms or seasonality. The new system uses a technique called Fast Fourier Transforms (FFTs) to identify underlying frequency in time series data, and then evaluates candidates against historical metric data to determine the best fit. Additionally, the ensemble algorithm selects the algorithm with the best performance, using a weighted approach that considers recent performance more heavily than older data. This allows New Relic to provide more accurate and effective alerts for its customers, without requiring them to do manual configuration or tuning.

New Relic has enhanced its Dynamic Baseline Alerts by implementing an auto-discovered seasonality system and an unsupervised ensemble algorithm to improve prediction accuracy for their clients. This development allows New Relic to automatically determine the seasonality of time-series data using Fast Fourier Transforms and selects the best-fitting algorithm from a set of options based on recency, trend, and seasonality factors. The ensemble system evaluates multiple algorithms, such as single and triple exponential smoothing, to find the optimal one using the MASE statistical method, often favoring simpler algorithms for data with minimal seasonality. This approach enables New Relic to manage over a billion events and metrics per minute for more than 15,000 customers, continually refining their systems to deliver real-time enhancements across their SaaS platform. Nadya Duke Boone, VP of Product Management at New Relic, emphasizes the company's commitment to leveraging data science and AI to support their customers' system management needs.

New Relic has experienced significant growth and complexity over the past few years, handling over 30 million HTTP requests per minute, 600 million new data points, and 50 billion events queried daily. To address this growth, New Relic's reliability practices have undergone a major overhaul, with key lessons learned from high-severity incidents in 2014 and 2015. The company has implemented various measures to improve reliability, including manual processes being automated, defining realistic metrics of reliability, making the right answer easy every time, not waiting for perfect answers, and ensuring autonomy within teams while maintaining organizational support. Through these changes, New Relic aims to create a culture that prioritizes reliability in large-scale systems.

New Relic has undergone significant growth, handling vast amounts of data and increasing in complexity, which has posed challenges in scaling its reliability practices. After a major incident in October 2014, the company recognized the need to improve its response to such events, leading to significant changes in its approach to reliability. Over fourteen months, New Relic implemented several initiatives, including creating email distribution lists, a Change Acceptance Board, and automating certain processes to reduce friction and improve incident management. The company also focused on refining its Mean Time to Resolution (MTTR) by improving processes and involving senior staff in on-call rotations. By defining concrete metrics for reliability and instituting a "Don't Repeat Incidents" policy, New Relic aimed to enhance stability and prevent recurrent issues. The introduction of a service maturity model and embedding Site Reliability Engineers in teams further strengthened their approach. Ultimately, New Relic's journey reflects an iterative process of learning and adaptation in managing reliability in complex systems, with plans to continue evolving as the company grows.

Sumo Logic and New Relic have announced an integration that allows users to combine the strengths of their SaaS services to provide a powerful and complete view of digital businesses, from client to infrastructure. The integration enables users to unlock insights by sending machine data analytics with application and infrastructure performance data into New Relic Insights via a custom webhook built directly into Sumo Logic. This allows users to visualize information such as events, alerts, and KPIs in a single pane of glass, providing an integrated context for faster root cause analysis and reduced Mean Time to Resolution (MTTR). The integration is achieved through three simple steps: configuring the New Relic webhook connection, scheduling a search to send custom events, and visualizing events in New Relic Insights.

Sumo Logic and New Relic have strengthened their partnership by integrating machine data analytics with application and infrastructure performance data through a custom New Relic webhook embedded in Sumo Logic. This integration allows users to monitor, alert, and visualize key events from applications and infrastructure within New Relic Insights, facilitating faster root cause analysis and reducing Mean Time to Resolution (MTTR) by offering a comprehensive view of digital business operations. The process involves configuring a New Relic webhook connection in Sumo Logic, scheduling searches to send custom events, and visualizing these events in New Relic Insights using NRQL. This seamless integration aims to enhance efficiency by reducing the need to switch between tools and enabling deeper analysis using Sumo Logic's machine learning and advanced analytics capabilities.

New Relic, a company that provides infrastructure monitoring services, has implemented various techniques to limit API overloading and protect itself from intentional or inadvertent abuse. One such technique is the separation of API and UI worker pools, which helps prevent API users from consuming all available worker time. Another technique is the use of an internal tool called API Overload Protection, which tracks worker time usage for each API key and restricts access if excessive usage is detected. However, New Relic recently encountered a puzzling issue where API requests were being reported as having used less time than they actually had, leading to inaccurate overload protection. The investigation revealed that the issue was caused by clients setting an aggressive timeout value on their requests, which caused nginx to give up on waiting for Unicorn's response before writing it back to the client. To fix this issue, New Relic set a configuration option in nginx to defer post-processing work until the upstream server had actually sent a response, ensuring that the API Overload Protection tool gets an accurate accounting of worker time consumed by each account.

APIs are increasingly essential for businesses to efficiently interact with customers and partners, but managing API usage to prevent overloading is a growing challenge. New Relic addresses this by employing various techniques to limit resource usage and mitigate overloads, including separating API and UI hardware resources and using Unicorn web servers for request handling. The company developed an API Overload Protection tool that tracks usage per API key and restricts access if usage becomes excessive. This tool operates independently of the application to protect against application failures, utilizing nginx to monitor API access and timing. A recent internal investigation revealed discrepancies between reported and actual API usage, attributed to client-imposed timeouts impacting nginx's logging and leading to inaccurate data reporting. The solution involved adjusting nginx's configuration to ensure more accurate tracking of API usage, highlighting the importance of robust monitoring systems as API use continues to grow.