December 2023 Summaries

At New Relic, we recently wrapped up our third annual 21 Days of Goodness campaign, a time for employees to give, donate, and share gratitude with their communities and each other. More than 75% of employees joined in, supporting over 1,100 charities worldwide with donations ranging from $25 to $500. The campaign also focused on giving time, with over 1,100 employees volunteering their time across 20 countries, including virtual and in-person opportunities. In the final week, employees shared what they were grateful for, recognizing each other's contributions and personal stories of gratitude. Overall, the campaign directed more than $130,000 to nonprofits worldwide, building on previous years' totals of over $335k.

New Relic's third annual 21 Days of Goodness campaign, which fosters community engagement and charitable giving, saw remarkable participation from over 75% of employees, supporting more than 1,100 charities worldwide. The campaign began on Giving Tuesday, when employees received funds in their Benevity accounts to donate to charities of their choice, with some receiving up to $500. This initiative was enhanced by personal donations and the company's donation matching program, ultimately directing over $130,000 to nonprofits globally. During the second week, the focus shifted to volunteering, with over 1,100 employees engaging with organizations across 20 countries, including partnerships with CareerVillage, MigraCode, and CodePath. The final week emphasized gratitude, encouraging employees to share appreciation for each other and their work. The campaign not only underscored New Relic's commitment to social impact but also strengthened employee connections through shared acts of service and acknowledgment.

OpenTelemetry is an open-source observability framework that can be daunting to instrument services with, but various tools are available to try it out without investing significant time in documentation. The otel-cli tool allows users to send OpenTelemetry traces to a collector or New Relic from the command line, enabling quick distributed tracing and adding tracing to shell scripts. telemetrygen is another utility that simulates a client generating traces, metrics, and logs, providing a basic way to see OpenTelemetry data without needing an application. The OpenTelemetry Community demo application offers a real-world example of a distributed system instrumented by OpenTelemetry, allowing users to view instrumentation examples, modify configuration files, and understand generated data. Additionally, New Relic provides examples in several languages for hands-on experimentation with OpenTelemetry instrumentation.

OpenTelemetry, an open-source observability framework officially launched in 2019, can present a steep learning curve for newcomers seeking to instrument their services. This blog post by Reese Lee, a Senior Developer Relations Engineer, offers several tools and methods to facilitate an easy trial of OpenTelemetry, allowing users to generate and analyze data before committing to full instrumentation. It introduces otel-cli, a command-line interface tool for sending OpenTelemetry traces to collectors or New Relic, as a straightforward option for distributed tracing. Additionally, telemetrygen is highlighted as a utility to simulate client-generated traces, metrics, and logs for testing and demonstration purposes, though its data is static and basic. For a more comprehensive experience, the OpenTelemetry Community demo application provides a real-world example of a distributed system, complete with microservices and a load generator to simulate traffic. New Relic also offers simple application examples in various languages to help users understand the instrumentation process. The blog emphasizes that the views expressed are the author's own and encourages readers to explore further resources and community support for a deeper understanding of OpenTelemetry and New Relic’s offerings.

When patients have serious or life-threatening health issues, making decisions in partnership with their families and healthcare professionals can be overwhelming. ACP Decisions empowers patients to make more informed medical decisions through easy-to-understand videos that are value-neutral and culturally appropriate. To address the challenge of video playback interruptions, ACP Decisions turned to New Relic for a telemetry platform to diagnose and predict performance issues in their technologies, especially to scale and optimize the clinician and patient experience. By setting up monitoring alerts and refining instrumentation, ACP Decisions aims to increase uptime and reliability, identify errors, and understand user experiences. The organization is still evolving its dashboards and prioritizing observability needs through a gap analysis, with plans to enhance alert monitoring and logging capabilities. Through their Observability for Good program, ACP Decisions supports nonprofits in driving equitable access to technology, offering free and discounted access to the New Relic platform and pro bono support.

ACP Decisions aims to empower patients in making informed medical decisions by providing culturally appropriate and value-neutral videos, yet faces challenges in maintaining seamless technology performance. To address potential disruptions in clinical workflows, they partnered with New Relic to enhance their observability and monitoring capabilities. This collaboration has allowed them to better diagnose and predict the system's performance, increasing uptime and reliability across their web and mobile platforms. The use of New Relic's tools, including NRQL, Lambda, and Docker monitoring, has facilitated an iterative process of improving their technological infrastructure. ACP Decisions also participates in the Observability for Good Social Impact program, offering free and discounted access to New Relic's platform to other nonprofits, aiming to improve equitable technology access.

At BiteSite, a small custom software agency, they use Ruby on Rails and React for web and mobile applications. They initially used Heroku's free plan but encountered performance issues due to server spinning down after 30 minutes of inactivity. To address this, they turned to New Relic, which helped them monitor availability and identify performance problems. With New Relic, they can track transactions automatically, providing actionable insights on where performance issues lie. This enables them to quickly implement solutions, validate changes, and improve overall application performance. The agency appreciates the value New Relic delivers, particularly for small businesses with limited resources. They use the free tier of New Relic, which provides a clear roadmap for fixing issues, resulting in immediate impact and improvement in their development process.

BiteSite, a small software agency specializing in custom web and mobile applications, has found significant value in using the New Relic platform for performance monitoring, particularly within their Ruby on Rails and React projects. Initially, they used New Relic as a creative solution to prevent Heroku's free server from spinning down, ensuring consistent availability. Over time, New Relic has become an essential tool for diagnosing and addressing performance issues, providing real-time insights into transaction steps and enabling quick, actionable solutions. The platform's user-friendly interface and immediate feedback loop have allowed BiteSite to efficiently optimize web application performance, enhance customer satisfaction, and validate their development efforts, all while maintaining a focus on Agile methodologies and minimum viable products. Despite their limited use due to resource constraints, the agency recognizes New Relic's enduring impact and clarity in delivering measurable improvements, underscoring its value for small businesses with constrained resources.

New Relic's observability platform handles pipeline latency by attempting to keep data in order, buffering and reordering data, dynamically adapting to latency with strategies like traffic flow, time waves, and external observers. The goal is to design a system where latency has less impact, while also addressing challenges such as blockages in the pipeline and lack of information for uninstrumented paths. By combining these strategies and scoping them appropriately, New Relic aims to dynamically adjust buffering and processing to improve pipeline performance and resilience.

In data-processing pipelines where timeliness is crucial, New Relic addresses unexpected latency using various strategies to ensure functional correctness and real-time observability. The company operates a large-scale streaming ETL pipeline that ingests, processes, and stores data, with customers depending on timely alerts for system behavior insights. To handle latency, New Relic implements techniques such as maintaining data order, buffering and reordering, and dynamically adapting buffering based on latency tracking. Strategies include using traffic flow metadata, synthetic "time waves" to detect lag, and external monitoring of pipeline segments. Each method has its challenges, like the difficulty in detecting blockages or ensuring coverage across all paths. By combining and scoping these strategies, New Relic can adjust data processing dynamically to minimize latency impact, which is vital for maintaining the accuracy and reliability of alerts and overall system performance.

Today's software is significantly more complex than it was 20+ years ago, making troubleshooting increasingly challenging. However, implementing observability into systems has helped in understanding application performance and identifying issues. Observability for CI/CD pipelines has not progressed as much as expected, despite their importance in the software delivery process. OpenTelemetry is an open-source observability framework that provides a set of APIs, SDKs, instrumentation libraries, and tools to help generate, collect, transform, and export telemetry data. It offers freedom from vendor lock-in and provides useful tools such as the Collector, which ingests, transforms, and exports data to one or more observability backends. Observability for CI/CD pipelines is crucial for troubleshooting issues, achieving development agility, and gaining insights into pipeline health. While OpenTelemetry is a promising solution, there are challenges in standardization, tooling landscape disorganization, and limited adoption by popular CI/CD tools.

The increasing complexity of modern software and the rapid evolution of development processes, particularly with the introduction of DevOps and CI/CD, have highlighted the need for improved observability of CI/CD pipelines to ensure efficient software delivery. Observability, defined as understanding a system from the outside by analyzing emitted data, is crucial for troubleshooting and maintaining healthy pipelines, which are essential for continuous integration and delivery of code into production. OpenTelemetry, an open-source framework, offers a standardized approach to collecting and exporting telemetry data, aiding in achieving pipeline observability without vendor lock-in. Despite its utility, the integration of OpenTelemetry into CI/CD tools is still in its nascent stages, and the tooling landscape remains fragmented, presenting challenges in standardizing observability practices across different platforms. Consequently, the development and maintenance of CI/CD observability tools rely heavily on community contributions, which poses potential risks if such tools become unsupported. Nevertheless, efforts continue to advance the observability capabilities of CI/CD pipelines, with companies like New Relic offering solutions for monitoring and analyzing pipeline data to enhance their performance and reliability.

Improving the end-user experience is crucial for business growth, as it directly impacts customer satisfaction, conversion rates, brand perception, and support costs. A well-designed user experience serves as a compass guiding businesses towards success by creating a digital environment that's both functional and enjoyable. By leveraging practical strategies such as optimizing website navigation, speeding up page load times, ensuring mobile responsiveness, personalizing user interactions, and focusing on user-centred design, businesses can create a more engaging and satisfying user experience. This approach leads to increased customer satisfaction, higher conversion rates, enhanced brand perception, reduced support costs, data-driven insights for improvement, and a competitive advantage. By prioritizing the end-user experience, businesses can drive growth, enhance customer loyalty, and stay competitive in the ever-evolving digital landscape.

Navigating the digital landscape successfully requires a seamless end-user experience, which acts as a crucial component in driving business growth and success. The end-user experience is not merely a superficial aspect; it incorporates usability, performance, accessibility, personalization, and consistency, all of which contribute to how users perceive and enjoy their interaction with digital products and services. Enhancing this experience leads to increased customer satisfaction, higher conversion rates, improved brand perception, reduced support costs, and competitive advantage, all of which significantly impact business growth. Practical strategies to improve user experience include optimizing website navigation, speeding up page load times, ensuring mobile responsiveness, personalizing user interactions, and focusing on user-centered design. Tools like New Relic can offer valuable insights into user interactions and performance, helping businesses optimize their digital platforms effectively. Overall, prioritizing the end-user experience is essential for staying competitive and achieving business success in the digital era.

With the rise of Large Language Models (LLMs), it's increasingly important to monitor their performance as they can be unpredictable and have varying outputs. Observability tools like OpenTelemetry can help analyze inputs and outputs of complex software, including LLMs, providing multiple signals needed for troubleshooting in production. However, LLMs are limited by the information they were trained on and cannot access or retrieve real-time data. Frameworks like LangChain extend their power by integrating them with external databases or APIs through retrieval-augmented generation (RAG). RAG allows models to first retrieve information from a database or search engine before generating a response, providing up-to-date and relevant information. OpenTelemetry can be used to collect and export metrics, logs, and traces from LLM applications, enabling tracing, error tracking, cost management, and A/B testing of performance. By leveraging these tools, developers can build reliable systems with guide rails, even with the use of unpredictable technologies like LLMs.

The blog post discusses the importance of observability in managing large language models (LLMs) in production, highlighting the unpredictable nature of these AI models and the necessity of tracking their performance through tools like OpenTelemetry. It explains how frameworks such as LangChain and LlamaIndex can extend LLM capabilities by integrating them with external databases or APIs via retrieval-augmented generation, allowing models to access real-time information. The article provides insights on debugging LangChain applications by using traces to identify performance issues and the benefits of manual instrumentation for detailed tracking. It also introduces OpenLLMetry, an open-source project for auto-instrumenting LLM apps, and emphasizes the need for prompt engineering and token management to optimize costs and outputs. Additionally, it covers strategies for A/B testing different LLM models to find the most cost-effective and efficient solutions while advocating for the application of traditional code maintenance practices to manage AI applications effectively.

Cluster management is a critical aspect of optimizing resource utilization in computing environments. It involves orchestrating nodes to work together efficiently, maximizing their potential while avoiding wastage or overload. Effective cluster management ensures performance, scalability, and reliability, directly impacting an organization's bottom line. Optimizing resource utilization is vital for enhancing operational efficiency and cost-effectiveness, as it allows organizations to reduce the need for additional hardware, leading to substantial savings in both capital and operational expenses. A well-managed cluster also translates to improved system performance and reliability, offering greater scalability and flexibility in rapidly changing business environments. Key components of cluster management include node management, load balancing, scalability, monitoring and maintenance, redundancy and failover strategies, and adding or modifying cluster elements. Adopting certain best practices, such as data-driven resource allocation, efficient workload distribution, scalability planning, regular performance optimization, and advanced resource monitoring, can significantly enhance the efficiency and reliability of cluster management.

Cluster management is crucial for optimizing the performance, scalability, and reliability of computing resources by effectively orchestrating a network of linked computers to work as a cohesive system. This involves key components such as node management, load balancing, scalability, monitoring, maintenance, and implementing redundancy and failover strategies. Efficient cluster management can lead to significant cost savings by optimizing resource utilization, reducing the need for additional hardware, and enhancing system performance and customer satisfaction. Moreover, Kubernetes has become the standard for container orchestration, offering robust tools for managing containerized applications at scale, including deployment, scaling, and monitoring. Best practices for cluster management emphasize data-driven resource allocation, efficient workload distribution, scalability planning, regular performance optimization, and advanced resource monitoring. Tools like New Relic provide comprehensive monitoring capabilities that facilitate these practices, ensuring a robust and responsive infrastructure in a competitive, technology-driven landscape.

The company Crisp transitioned from multiple open-source monitoring tools to a single observability tool, New Relic, due to complexity and cost concerns. They initially chose an observability provider based on its pricing plan, but were still stuck with multiple tools after a year. After revisiting their strategy, they switched to New Relic, which offered a predictable pricing model, streamlined operations, and improved the return on investment from their monitoring tool. With New Relic, they were able to onboard new engineers in six months, turning their technical operations team into a center of excellence, and improve mean time to recovery (MTTR) by 95%. The company aims to further optimize their workflows with business-as-usual ratios and TerraForm.

Crisp, a risk intelligence company, transitioned from a complex array of open-source monitoring tools to a single observability platform by adopting New Relic. Initially, Crisp struggled with unpredictable costs and the complexity of managing various tools during their shift to a Google cloud platform. After revisiting their strategy, they chose New Relic for its predictable pricing model and comprehensive support, which includes New Relic University and extensive documentation. This switch allowed Crisp to streamline operations, reduce the mean time to recovery by 95%, and foster a culture shift towards more proactive monitoring. New Relic's real-time data dashboards enable engineers to quickly identify and address issues, improving workflow efficiency and reducing burnout. Moreover, Crisp's engineers benefit from a faster onboarding process and are encouraged to engage with ongoing training. The company aims to further optimize its business-as-usual metrics to focus more on project-related monitoring, leveraging New Relic's capabilities to enhance risk intelligence for their clients.

In the fast-moving world of modern software development, errors are inevitable and can have a significant impact on customers. New Relic's recent enhancements to its error inbox aim to mitigate this by providing effective error tracking integrated with observability data, allowing developers to identify and track errors in real-time. The updates include custom fingerprinting to group similar errors, tracking errors by version, and unifying performance data and errors in a single UI, making it easier for developers to streamline their debugging processes, enhance code quality, and turn errors into opportunities for growth.

In the evolving landscape of software development, effective error tracking is crucial to managing the complexity and potential impact of errors on customer experience. New Relic has introduced significant enhancements to its errors inbox, which streamline the debugging process and reduce downtime by grouping similar errors through custom fingerprinting and tracking errors by version. This allows for a focused approach to error resolution, minimizing alert fatigue and enhancing the ability to identify the source of errors in fast-paced development environments. The integration of New Relic APM 360 provides a unified view of performance, change, security, and error data, which accelerates triage and improves overall error management. These advancements offer opportunities for improving application quality and customer satisfaction by turning errors into avenues for growth.

New Relic AI aims to unlock observability for all by enabling users to interact with their telemetry data using natural language, reducing reliance on New Relic Query Language (NRQL) proficiency and streamlining analysis across tech stacks. To evaluate the performance of this system, the development team identified areas where clear-cut measures of success can be applied, such as classifying user requests into different flows and performing syntax validation in NL2NRQL. They also tackled hallucinations by using retrieval-augmented generation (RAG) and reframing the task to ensure the model relies on external context rather than built-in knowledge. Additionally, they promote trustworthiness via transparency and clear communication throughout the Q&A flow, providing users with intermediate steps and explanations to assess the validity of answers. By implementing these strategies, New Relic AI can deliver real value to users while minimizing potential pitfalls.

The New Relic browser agent is a tool that helps monitor the performance, errors, and user experience of web applications and sites by collecting data on page load time, Ajax requests, JavaScript errors, and custom events, which can also be used to observe real user monitoring (RUM) data in web extension views. To use it in a web extension view, developers need to follow specific steps, including installing the browser agent as an NPM package and loading it into their extension view code. Once installed, the agent can detect RUM data and provide insights for optimizing and troubleshooting the web extension's performance and user experience.

New Relic AI was developed to enhance observability by allowing users to interact with telemetry data through natural language, reducing the dependence on New Relic Query Language (NRQL) proficiency. The AI aims to provide accurate insights, identify system anomalies, and streamline tech stack analysis, while addressing challenges like performance validation and AI hallucinations. The team implemented strategies such as modular design for systematic evaluation, syntax validation for NRQL queries, and retrieval-augmented generation (RAG) to mitigate hallucinations and enhance response accuracy. Transparency and clear communication are promoted by providing users with context and intermediate steps in the Q&A process, enabling them to evaluate the AI's answers. User feedback is integral to improving New Relic AI's capabilities, with ongoing input collected to refine and enhance its performance.

The New Relic browser agent is a tool designed to monitor the performance, errors, and user experience of web applications, including views rendered by web extensions like pop-ups, sidebars, or options pages. By installing the browser agent NPM package into a web extension project, users can collect real user monitoring (RUM) data such as page load times, Ajax requests, and JavaScript errors, which are then sent to New Relic for analysis. The installation involves adding the agent as a dependency using NPM and configuring it within the web extension code to ensure it properly collects and records data. This setup allows developers to view performance data in the New Relic dashboard, aiding in the optimization and troubleshooting of web extensions, while the blog emphasizes that the solutions presented are environment-specific and not part of New Relic's commercial offerings.