January 2020 Summaries

The DevOps and IT Operations landscape is evolving rapidly, with increasing pressure to ship software faster, more frequently, and with greater reliability. However, this shift brings new challenges, including a wider surface area to monitor and react to, increased data volume, and response fatigue. To address these issues, AIOps (Artificial Intelligence for IT Operations) has emerged as a technology category that puts AI and machine learning in the hands of on-call teams to prevent more incidents and respond to them faster. AIOps platforms analyze data generated by software systems to predict possible problems, determine root causes, and drive automation to fix them, enhancing monitoring, service management, and automation tasks. By providing an intelligent feed of incident information alongside telemetry data, AIOps augments the value of monitoring, enabling teams to pinpoint issues faster, understand why they occurred, and take proactive action. The use cases for AIOps include proactive anomaly detection, event correlation and noise reduction, intelligent alerting and escalation, and automated incident remediation, which can help DevOps, SRE, and on-call teams detect problems before they cause outages or performance issues, diagnose incidents more efficiently, and resolve them faster.

New Relic Connect is a customizable system that allows SRE and DevOps teams to integrate with additional notification platforms, modify webhook payloads with new attributes, and enrich webhooks with information from NRDB. It can be used to easily integrate New Relic Alerts with any Amazon Web Services (AWS) offering, such as sending incident data to Amazon EventBridge. Connect enhances the power and flexibility of an existing webhook's functionality in New Relic Alerts by allowing users to execute NRQL queries, map attributes, and send webhooks to new destinations. By using Connect with EventBridge, teams can prevent disruptions to their services' functionality and uptime, automate scaling applications, respond to infrastructure anomalies, and gather more information based on custom New Relic Insights events.

In today’s marketplace, digital commerce drives substantial competition and revenue growth for organizations of all sizes, with over $500 billion generated in the U.S. alone during peak events like holiday shopping. However, if a company's technology stack fails during a critical event, it can be costly, highlighting the need for rapid incident response and resolution. New Relic is introducing real-time streaming for APM events to enable DevOps and SRE teams to respond immediately to errors and anomalies, providing real-time visibility into application performance data. With this feature, teams can move faster with confidence, rapidly recover from incidents, and react to demand surges and business metric changes. Real-time streaming is available at no additional cost for all APM agents, and getting started requires updating the APM agents to the latest version and adding a specific query clause to charts. This feature is currently only available for APM events, but it can help ensure teams are fully prepared for peak events, deployments, or incidents.

Digital commerce significantly contributes to organizational competition and revenue, with U.S. digital retail surpassing $500 billion in 2018 and events like Cyber Monday 2019 hitting record sales. To maintain seamless customer experiences during peak events, it is crucial for DevOps and SRE teams to rapidly address software issues. New Relic has introduced real-time streaming for Application Performance Monitoring (APM), providing data updates every five seconds, which is 12 times faster than before. This enhancement enables teams to respond swiftly to incidents, anomalies, and demand surges, especially during critical business periods. Real-time streaming is available for all APM agents at no extra cost and can be accessed by updating to the latest version. The feature allows for near-instantaneous data visibility in New Relic One dashboards, although it currently applies only to APM events.

The author of this text is a Chief People Officer at New Relic, with experience in management consulting for over 15 years. They believe that the key to creating an environment where employees are excited to show up every morning and do their best work lies in three main things: providing a clear and exciting mission, helping employees learn and grow, and fostering a sense of belonging. The author emphasizes the importance of connecting people's work to a broader, meaningful mission, helping employees learn and grow through strategic promotions and career development opportunities, and creating an environment that fosters a sense of community and belonging. By doing so, leaders can create a positive and productive work culture that benefits both employees and the company as a whole.

Kristy Friedrichs, New Relic's Chief People Officer, shares insights on creating a thriving workplace culture by focusing on three main strategies: connecting employees’ work to a meaningful mission, facilitating learning and growth opportunities, and fostering a sense of belonging. She emphasizes the power of having a higher purpose in motivating employees, as demonstrated by New Relic's mission to enhance software and digital experiences. Friedrichs advocates for proactive career development, illustrated by the company's strategic internal mobility initiatives that have led to numerous promotions and departmental changes, particularly from tech support roles. Additionally, she highlights the importance of creating a diverse and inclusive environment where employees feel safe to be their authentic selves, which begins with recruiting individuals who can contribute fresh perspectives to the culture.

If you’ve ever been an on-call SRE, you’re familiar with alert fatigue: the burned out feeling that creeps in after responding to alert after alert from tons of services and tools across your stack. Not only is this phenomenon exhausting, but constant pages also limit your ability to focus on other work, even if you’re simply clicking “acknowledge” (“acking”). Research has shown that people lose up to 40% of productive time with brief context switches. Many of the alerts causing never-ending streams of pages are neither urgent nor important, and don’t require any human action. These notifications come from all kinds of tools in your production system and tend to get quickly acked but largely ignored since there usually isn’t an underlying actionable issue. Low-priority alerts indicate problems that may eventually need to be addressed, but are low on the current priority list. Flapping alerts can feel like playing whack-a-mole, with unrelated issues sometimes getting lost in piles of notifications. Duplicate alerts and correlated alerts, such as those from redundant monitoring configuration or complex system issues, require more investigation time and can build frustration. Implementing an AIOps platform like New Relic AI can help tackle alert noise across your stack by providing machine learning-driven filters and logic to correlate and prioritize incidents, reducing pager fatigue and empowering teams to stay focused on important issues.

Alert fatigue is a common issue for on-call Site Reliability Engineers (SREs) and DevOps teams, resulting from the continuous influx of notifications that are often neither urgent nor actionable, leading to productivity loss due to frequent context switching. These alerts often stem from irrelevant notifications, low-priority issues, flapping and duplicate alerts, and correlated alerts that lack sufficient context to identify root causes. Addressing this noise is crucial, as it can obscure genuinely critical problems within complex production systems. New Relic's AIOps platform, still in private beta, offers a solution by utilizing machine learning-driven filters to streamline alerts, enhance incident correlation, and prioritize issues, thereby reducing pager fatigue and allowing teams to concentrate on essential tasks. Guy Fighel, the General Manager of Applied Intelligence at New Relic, emphasizes that while these insights reflect his personal views, they aim to provide guidance on improving alert management in dynamic environments.

Breadcrumbs can be incredibly useful for mobile app developers to gather specific information at designated points in their app, build a story around poor performance, and understand user flow through an app. By adding breadcrumbs to key workflows, such as login processes or shopping cart flows, developers can make troubleshooting much easier after their app is released. This can help identify issues like frequent token refreshes, improve app store reviews, and decrease support tickets. With the use of mobile breadcrumb custom events in New Relic, developers can correlate information between session data and breadcrumbs to build performance stories and gain insights into user behavior.

Breadcrumbs in mobile applications serve as a valuable tool for developers to identify and troubleshoot performance issues by providing a trail of user actions, particularly in areas like login processes and shopping cart flows. The post describes how New Relic's Mobile Apps team utilized breadcrumbs to address login issues by placing them strategically throughout their apps to maintain consistency and identify problems, such as the frequent refreshing of user tokens, which led to improved app store reviews and reduced support tickets. Breadcrumbs, captured via the New Relic Mobile event crash trail, allow developers to monitor user flow and correlate session data, providing insights into user interactions and application performance. This data can be analyzed using the New Relic Query Language (NRQL) to generate informative dashboards. The author emphasizes the importance of monitoring client-side performance and using tools like New Relic Browser, Synthetics, and Mobile to enhance the digital experience.

The benefits of enterprise software investments in business history have led to significant productivity gains for U.S. workers, with global enterprises expected to spend nearly $1.2 trillion on technology tools in 2019 to digitally transform themselves. However, measuring the return on investment is often challenging due to the lack of quantifiable benefits and the need to make a compelling business case to higher-ups who sign the checks. Observability software like New Relic can play a pivotal role in digital transformation and driving customer experience, but its value proposition may not be immediately obvious. By consolidating accounts, sharing data, monitoring key metrics that customers care about, aligning KPIs with business value, factoring in engineer productivity, getting dev and ops on the same page, and getting buy-in at the top, companies can unlock the true potential of observability software and drive meaningful business outcomes.

Enterprise software has historically been a significant investment for businesses, enhancing productivity and driving digital transformation, yet measuring its return on investment can be challenging. Observability software, like New Relic's, plays a crucial role in this transformation by helping companies understand and improve their services, despite often being undervalued compared to more apparent tools like CRM systems. In a case study of a major tech client, New Relic demonstrated the value of its software by consolidating accounts, encouraging data sharing, updating monitoring metrics to reflect customer concerns, aligning KPIs with business outcomes, and highlighting productivity gains among engineers. The company emphasized the importance of getting buy-in from top executives by translating technical benefits into financial terms, thereby underscoring the broader impact of observability on the customer experience and business performance.

New Relic Logs helps teams simplify log management and troubleshoot issues more efficiently by combining log data with application performance and infrastructure health in one place. With logs in context, users can easily correlate log data with events and errors across their stack, reducing mean time to resolution (MTTR) and improving overall troubleshooting efficiency. The new feature is available for all New Relic APM agents, including support for multiple programming languages such as Go, Java, .NET, Node.js, PHP, Python, and Ruby. By linking log data directly to applications, APM errors, and instrumented traces, users can quickly identify meaningful trends in their data and narrow down the root cause of issues.

As software teams increasingly modernize their applications in the cloud, the volume of log data they manage grows, leading to challenges in maintaining context during troubleshooting. New Relic Logs addresses this by integrating log data with application performance and infrastructure health, allowing teams to correlate logs with events and errors, thereby reducing the mean time to resolution (MTTR). The tool supports multiple programming languages and environments, such as Kubernetes and AWS Lambda, enabling log data integration across diverse applications. By using logs in context, users can link log data to application traces and errors, facilitating quicker identification of trends and issues. This integration is further enhanced by pre-built plugins for popular log forwarders, making data forwarding to New Relic seamless. New Relic encourages users to explore these capabilities within its platform, providing resources and support through its Explorer's Hub community.

New Relic Alerts` is a tool that helps DevOps teams ensure they can respond quickly to service issues by automatically notifying the right team members when predefined alert conditions are met. To avoid `alert fatigue`, teams need to manage alert policies and conditions carefully, focusing on key metrics while filtering out expected behavior. The post provides examples of best practices for setting up alert conditions for different types of applications and hosts monitored with New Relic Browser, APM 360, and Infrastructure. It also emphasizes the importance of understanding thresholds and baselines in creating effective alerting strategies.

Alerting is a critical practice for successful DevOps teams, and New Relic Alerts provides a system to ensure timely notifications to the appropriate team members when monitored applications, hosts, or entities encounter issues. The platform allows teams to manage alert policies and conditions effectively, focusing on key metrics while minimizing alert fatigue. Suggestions for setting alert conditions are offered, based on field best practices, for applications monitored by New Relic Browser, New Relic APM, and New Relic Infrastructure. These include establishing threshold and baseline conditions to detect deviations in metrics like page load time, transaction throughput, error rates, and host health. The alerting strategy is essential for modern, dynamic, and scaled environments, and the blog emphasizes the importance of designing and maintaining an alerting system suited to organizational needs. The views expressed are those of the author, Eric Mittelhammer, and do not necessarily reflect New Relic's stance, with the blog encouraging readers to explore further support and discussions on the Explorers Hub.

New Relic has expanded its paid parental leave policy to 12 weeks of full pay for employees following a child's birth, adoption, or placement, effective January 1, 2020, and also introduced backup childcare support through Care.com. This enhancement aims to provide more family bonding time and places New Relic at the forefront of a rapidly changing competitive landscape. The company is committed to exploring opportunities to extend similar support for parents and caregivers outside the United States. With U.S. unemployment at historic lows, excellent benefits like paid parental leave are crucial in attracting and retaining employees, reducing turnover costs. The lack of mandated paid parental leave in the United States is a significant consideration, with only eight states having passed state paid family leave laws, and no federal law requiring private-sector employers to provide paid leave. New Relic's commitment to providing excellent employee benefits aligns with growing public and private sector momentum to support working families.

New Relic has expanded its employee benefits, notably enhancing its paid parental leave from 10 to 12 weeks for new parents globally, effective January 1, 2020, alongside up to 10 weeks of paid disability leave for birthing parents. This policy includes both base and variable pay, setting New Relic apart in the competitive landscape. Additionally, U.S. employees now have access to 10 days of backup childcare through Care.com, which can be used when regular childcare arrangements fail. The expansion of these benefits aligns with broader trends in the U.S., where the Federal Employee Paid Leave Act now grants 12 weeks of paid leave for federal civilian workers, although no federal mandate exists for the private sector. Currently, only eight states and the District of Columbia have state laws for paid family leave. New Relic's gender-neutral policy aims to promote equality, and highlights the company's commitment to supporting its employees and attracting talent in a low-unemployment market. The blog post also notes a growing public and private sector interest in supporting working families, with calls for federal legislation to ensure paid family and medical leave for more Americans.

The New Relic One blog series highlights popular open-source applications that can be deployed to gain specific insights into application performance. These applications include Event Stream, Neon, Network Telemetry, and SLO/R, each addressing different needs such as real-time data analysis, dashboard creation, network traffic visualization, and service-level objective (SLO) attainment measurement. To deploy these apps, users need access to New Relic One, the New Relic CLI, a personal API key, Node v10 or higher, and a GitHub account. The applications are designed to be customizable and can be forked or deployed from GitHub, with resources available for building custom applications as well as contributing to open-source projects.

New Relic One offers a programmable platform that allows companies to build custom applications tailored to their specific technology stacks and insights needs, exemplified through a series of open-source apps such as Event Stream, Neon, Network Telemetry, and SLO/R. These applications, which can be deployed from GitHub, cater to different monitoring and data visualization needs, such as live streaming application performance data, creating business unit dashboards, visualizing network traffic, and measuring service-level objectives. Developers are encouraged to use these apps or create their own using New Relic's resources, including its CLI, SDK, and GraphQL API. The platform also supports community contributions to its open-source projects, fostering a collaborative developer environment. Access to New Relic One requires a paid account, and developers can deploy applications using provided commands, with additional support and training available through New Relic's documentation and community forums.

New Relic Browser is the most deployed real user monitoring (RUM) tool in the world, providing quick context for developers to solve problems in modern software stacks. It helps businesses understand their end-user experience and relationship between website performance and technology stack health. With its dominance in the RUM market, New Relic Browser is deployed on nearly 1.8 million websites, over 20 times more sites than comparable monitoring solutions. Its popularity stems from its built-in support for modern web experiences, integration with the larger New Relic platform, ease of setup, and immediate time to value. Key features include SPA handling, user-centric performance metrics, AJAX monitoring, JavaScript error monitoring, and custom attribute support. The tool integrates seamlessly with the rest of the New Relic platform, allowing users to navigate from real user monitoring to application layer to infrastructure layer easily. Browser is also easy to set up and provides immediate value, making it a critical component of successful digital business strategies.

New Relic Browser is a prominent real user monitoring (RUM) tool designed to provide developers and software engineers with insights into end-user experiences by tracking interactions and performance metrics such as first paint and AJAX requests. Known for its ease of deployment and integration with the broader New Relic platform, it offers features tailored for modern web environments, including support for single-page applications (SPAs) and JavaScript error monitoring. Deployed on nearly 1.8 million websites, New Relic Browser distinguishes itself through its immediate time to value and its ability to enhance business understanding of user interactions from frontend to backend, making it a critical component of a successful digital business strategy.