October 2022 Summaries

New Relic Data Plus offers powerful observability capabilities to enhance security, scale, and performance. It provides features like data export, extended data retention, logs obfuscation, FedRAMP for enabled accounts, longer-running queries, and more. Historical data exports allow users to send existing data past initial ingestion out of New Relic, offering unlimited query durations and up to 200 million data point rows. Streaming data exports, on the other hand, send data exclusively at ingestion by New Relic, allowing users to explore data in real-time. Users can export historical data using a JSON file generated via NerdGraph, while streaming data exports require setting rules via NerdGraph about specific types of data to be exported.

New Relic Data Plus offers enhanced observability features, including the ability to export historical and real-time streaming telemetry data to external sources, making it valuable for compliance and data analysis needs. Users can choose between historical data exports, which allow for exporting data older than 12 hours in large quantities to analyze past events and fulfill compliance requirements, and streaming data exports, which continuously export data at the time of ingestion for real-time analysis and long-term retention. The historical data export process involves creating queries in NerdGraph and exporting data as JSON files, while streaming data export requires setting up custom NRQL-based rules via NerdGraph to govern the types of data being exported, with the flexibility to enable or disable these rules as needed. Data Plus also includes features like extended data retention, logs obfuscation, and FedRAMP for enhanced security, scale, and performance, supporting users in maximizing the observability and utility of their telemetry data.

At New Relic, the company aims to make observability a daily practice for all engineering teams to deliver better customer experiences by democratizing access to monitoring and debugging tools. The New Relic for Startups program has rapidly grown to include early-stage startups across various industries, with customers like Hex, which uses New Relic to monitor its large Kubernetes clusters and improve performance. To further their goal, New Relic is collaborating with Carta and Zendesk to provide a comprehensive ecosystem of tools to scale both engineering and business practices. Eligible startups can access the New Relic observability platform, as well as generous credits and discounts for Carta's equity management solutions and Zendesk's customer service solution, making it easier for them to focus on building companies around big ideas.

The Jenkins OpenTelemetry plugin enables monitoring and observability of Jenkins jobs and pipelines, allowing developers to shift left and improve their DevOps practices. The plugin integrates with New Relic, providing visualizations of key metrics such as job duration and error rates, distributed tracing, and alerts for pipeline issues. To set up the plugin, users need to install Jenkins and configure the OpenTelemetry settings, including authentication and OTLP endpoint. Once configured, users can send data from their Jenkins pipelines to New Relic, providing visibility into the build and deployment phases of the DevOps lifecycle. With this integration, developers can gain insights into their pipeline performance and make data-driven decisions to improve their development workflows.

New Relic aims to democratize observability by making it a routine practice for engineering teams through its New Relic for Startups program, launched in 2021. This initiative provides startups with the necessary tools to monitor, debug, and optimize their software stacks, enhancing customer experiences and fostering confidence in software delivery. The program has expanded rapidly across various industries, and companies like Hex have benefited from New Relic's capabilities in monitoring large Kubernetes clusters. To further support startups, New Relic announced collaborations with Carta and Zendesk, offering startups access to premium platforms that aid in scaling engineering and business operations. This partnership allows startups to use New Relic, Carta, and Zendesk's tools, providing access to the same developer resources utilized by leading engineering organizations, alongside generous credits and discounts. The collaboration seeks to enable startups to develop robust DevOps and data-driven engineering practices, ultimately helping them to build innovative companies while managing costs effectively.

Jenkins, a widely-used open-source automation tool, plays a crucial role in implementing continuous integration (CI) and continuous development (CD) workflows, which are essential for accelerating application deployment while reducing bugs. To enhance the reliability of these processes, monitoring not only the applications but also the CI and CD tools themselves is vital, which can be achieved through the Jenkins OpenTelemetry plugin integrated with New Relic's quickstart dashboard. This setup allows users to visualize key metrics, employ distributed tracing to identify pipeline issues, and create alerts for specific conditions, ultimately aiding in the "shift left" approach for more efficient DevOps practices. By sending Jenkins pipeline data to New Relic, developers can gain comprehensive insights into their DevOps lifecycle, improving observability during the build and deployment phases.

When deploying applications in the cloud, linking performance and availability to cloud resources is crucial for getting quick insights and making data-driven decisions. A new integration with Terraform has been added to aid in setting up New Relic cloud integrations, allowing users to configure and scale their observability more quickly and easily. With this integration, users can automate their configuration using a scripting language called HCL, and fine-tune their setup without manual operations. The integration is available for multiple cloud providers, including AWS, Azure, and GCP, and can be accessed through the Terraform documentation site or by reaching out to the Explorers Hub for support.

Integrating New Relic with Terraform streamlines the process of linking application performance and availability to cloud resources, facilitating quicker issue resolution, cost savings, and informed architectural decisions. Previously, setting up such integrations required extensive manual configuration, but Terraform simplifies this by allowing users to define their desired setup using its scripting language, HCL, which is then executed through New Relic APIs. This automation not only accelerates the scaling and modification of cloud configurations but also reduces the operational overhead associated with manual processes. The New Relic Terraform provider offers multiple resources to configure cloud integrations, with examples and guidance available in the Terraform documentation. Users seeking to enhance their observability setups can access tutorials, examples, and community support through New Relic's platforms, while additional insights and engineering tips are available on the Data Bytes YouTube channel.

AWS Lambda is a serverless event-driven service that automatically executes code to deliver the right AWS resources for your infrastructure without management, paying a fee for the compute time used. To get detailed performance data and avoid overpaying, continuous monitoring is necessary, which can be complex given the distributed nature of Lambda functions. Memory usage affects Lambda performance, with allocating too much memory leading to throttling and poor concurrency limits. Common issues include large functions, cold starts, execution time limits, and concurrency limits. To troubleshoot and improve Lambda performance, a third-party continuous monitoring solution is necessary, providing detailed information such as invocation criteria, error analysis, and a single view of all Lambda functions in every AWS region. New Relic offers full Lambda monitoring with features like invocation tracking, events, distributed tracing, logs in context, and inventoried tags and metadata, to help optimize performance and costs.

AWS Lambda offers a serverless, event-driven service that simplifies infrastructure management by executing code automatically, allowing companies to focus on scaling and prototyping without managing servers. Users pay for compute time, thus requiring efficient monitoring to avoid unnecessary costs. Memory allocation significantly impacts Lambda performance, with potential issues like cold starts, execution time limits, and concurrency limits affecting efficiency. Continuous monitoring, beyond AWS's basic tools, is vital for detailed insights into function behaviors, helping troubleshoot and optimize performance. New Relic provides comprehensive Lambda monitoring, offering detailed invocation data, event information, distributed tracing, and contextual logs, enhancing operational efficiency and cost-effectiveness.

The analysis compares the cost-effectiveness of New Relic's usage-based pricing approach to the traditional subscription-based models of observability platforms like Datadog and Dynatrace across small, midsize, and large engineering teams. The findings suggest that New Relic can offer up to five times more value for money for small teams, and three times more for midsize and large teams, allowing customers to add more users and data without increasing costs. The study uses publicly available pricing to demonstrate that New Relic’s platform enables more data instrumentation and improved outcomes for similar or lower budgets. The analysis highlights that New Relic's approach can enhance platform efficacy, customer experience, and reduce churn by offering more value without additional financial outlay, and emphasizes transparency by providing a detailed cost breakdown based on different team sizes and tech stack configurations.

New Relic has announced the public preview of its Security RX solution, offering a comprehensive approach to software security and risk management. This new tool, available with a free 90-day trial, aims to enhance security by enabling instant vulnerability detection across all application dependencies without additional configuration. Security RX integrates with existing security assessment tools like Snyk, GitHub Dependabot, and AWS Security Hub, providing a unified view of vulnerabilities, performance, and availability issues. It facilitates collaboration across teams by linking vulnerabilities to specific organizational contexts and assigning issues for resolution. Additionally, the platform supports a wide range of programming languages and provides recommendations for library updates and infrastructure upgrades based on known vulnerabilities. With future enhancements planned, including advanced testing tools, New Relic Security RX positions itself as a crucial component in modern observability stacks to mitigate security risks effectively.

The New Relic Roku agent provides comprehensive observability for streaming applications on the Roku platform, enabling engineers to collect telemetry data, measure network connectivity, viewer accessibility, and video quality, and gain instant visibility into performance with interactive data visualizations. With this solution, developers can analyze large quantities of data, identify critical bottlenecks, and pinpoint impacted services, ultimately reducing subscription churn, increasing revenue, and maintaining a positive brand image. The agent offers application and device-level analytics, viewer journey tracking, and preconfigured dashboards to help engineers detect and resolve issues such as channel crashes, HTTP request errors, latency issues, and rebuffering, providing a wide range of functionality to address blindspots in Roku streaming applications.

New Relic has introduced the first comprehensive observability solution for Roku streaming applications through its Roku agent, providing engineers with enhanced visibility to optimize app and device performance. This solution allows for the collection and analysis of telemetry data to improve network connectivity, viewer accessibility, and video quality, which is crucial given Roku's significant market presence and the high risk of subscription churn. The New Relic Roku agent offers application and device-level analytics, including HTTP requests, errors, and video quality of experience metrics, which help diagnose issues more effectively. Additionally, it features a metrics API for aggregate analysis and preconfigured, interactive data visualizations, enabling quicker identification and resolution of OTT streaming issues to enhance user experience and reduce churn. By centralizing Roku performance data, developers can better maintain a positive brand image and increase revenue potential.

Amazon Elastic Compute Cloud (EC2) is a popular cloud computing system used in the AWS cloud, offering on-demand scalability and allowing users to launch virtual servers as needed. However, basic monitoring capabilities provided by Amazon CloudWatch are limited, failing to offer insights into application performance, processes, configuration, server OS, and application latency and error rates. To gain full observability of EC2 instances, users need to supplement CloudWatch with an observability platform like New Relic, which provides tools such as application performance monitoring (APM), real user monitoring, synthetic monitoring, and more. This allows users to troubleshoot issues in their applications and infrastructure, choose the right size and kind of EC2 instance for their services, and optimize resource allocation to avoid unnecessary cloud spend. By setting up a monitoring plan, including defining service level objectives (SLOs), determining who will monitor, notify, and report on issues, and selecting relevant tools, users can effectively manage their EC2 instances and ensure they are using the right resources. New Relic provides several integrations with AWS services, including automatic instrumentation of EC2 instances, integration with Amazon CloudWatch Metric Streams, and an Amazon EC2 quickstart for easy setup.

Parsing complex logs can be a challenge in the industry, especially when dealing with large semi-structured text fields that have historically been difficult to handle using New Relic Query Language (NRQL). NRQL variables allow you to assign a value computed from a non-aggregate function or attribute to an identifier that can be referenced elsewhere in the query. Regex multi-capture and the If function provide additional ways to simplify queries by reducing complexity and improving performance. Anchor parse, a new feature, leverages a syntax similar to LIKE to capture values, further simplifying queries. These features can help you get concise and faster insights into how your applications are performing, making it easier to read and write queries, and producing vastly faster results.

Parsing and understanding logs in New Relic has historically been challenging due to the complexity of semi-structured text fields, but recent advancements in NRQL, including variables, regex multi-capture, and anchor parse, have significantly streamlined this process. These new features enhance query efficiency and readability by simplifying the extraction and manipulation of data from logs, such as calculating the 99th percentile of response times with mixed units. NRQL variables allow for assigning computed values, regex multi-capture reduces the need for multiple regexes by capturing several values at once, and the If function introduces conditional logic directly into queries. The AnchorParse function further optimizes performance by providing a simplified syntax for capturing values. Collectively, these improvements reduce computational costs and increase the speed of data querying, offering a more effective way to gain insights into application performance.

In today's fast-paced development environment, achieving high code quality is crucial for innovation and growth. A development quality program can help track key performance indicators (KPIs) such as build success, unit test success, code coverage, and defect volume to identify areas that require improvement. By measuring code velocity and tracking data through custom dashboards, teams can establish a baseline, pinpoint areas of improvement, and focus their efforts on enhancing those modules with the highest defect volumes. This approach enables organizations to optimize their development processes, reduce outages, and improve overall quality.

A development quality program can enhance innovation and growth by improving code quality, which is crucial for reducing feature release failures and accelerating market delivery. The 2022 observability report indicates a strong relationship between full-stack observability and improved metrics such as fewer outages and faster detection and resolution times. Four crucial steps to achieving better code quality include tracking quality metrics like build success and defect volume, measuring code velocity to balance speed and stability, building development-quality dashboards with tools like New Relic, and establishing a baseline for continuous improvement. By focusing on these areas, development teams can identify defect sources, optimize processes, and ultimately improve their software's reliability and performance. Additionally, resources such as the release quality implementation guide, observability maturity guide, and New Relic's blog posts can provide further insights and support for enhancing code quality and operational efficiency.

New Relic introduced a usage-based pricing model with its unified platform for technology stack observability, diverging from traditional subscription pricing models by allowing customers to pay only for the data they use and the number of users accessing the platform. This approach has been well-received by customers like Chegg and ZenHub, who benefit from cost savings and comprehensive access to telemetry data without the constraints of traditional licensing models. New Relic's pricing model simplifies observability costs by focusing on data collection and user interaction, with a competitive rate of $0.35 per GB and options for free and full user access. This model addresses industry challenges of overpaying or underutilizing services, offering predictability and flexibility without penalties for increased usage. The platform's continuous feature enhancements are included in the pricing, which contrasts with other vendors that charge extra for new capabilities. The model's success is supported by positive feedback from industry analysts and comparisons illustrating its cost-effectiveness against other observability solutions.

This is a success story from the New Relic team about their experience with load testing and diagnosing memory usage issues in an Elixir-based API gateway. The team conducted thorough load testing to understand how their BEAM VM handled stress, and after two years of observing and learning, they were able to reproduce the issue and identify its root cause: a large number of processes created for SSL connections that held redundant data. They discovered that providing certificates to each individual process was causing this redundancy, and by using an ETS table to store the CA certs in one place, they reduced memory usage by 60-70%. This experience highlights the importance of understanding how tools like BEAM VM can fail and the value of diagnosing and resolving lower-level issues. The team's expertise in Elixir and Erlang allowed them to leverage powerful tools like the Erlang Observer to troubleshoot the issue and make a significant impact on their system's performance.

New Relic has extensively used Elixir and Erlang's BEAM VM for its high-throughput services, particularly in the New Relic Unified API team, where they focus on understanding both the strengths and potential failure scenarios of these tools. Through rigorous load testing and capacity analysis, they identified memory spikes during requests to downstream services, often caused by a large number of HTTP requests, which sometimes led to out-of-memory (OOM) conditions. Investigating further, they utilized the Erlang Observer to monitor memory usage and discovered that redundant memory allocation in SSL connection processes was the issue, exacerbated by binary DER blobs used for SSL certificates. By switching to an ETS table for certificate storage, they significantly reduced memory usage by 60-70% during expensive queries and improved the system's overall efficiency. This exploration enhanced their proficiency in diagnosing and resolving lower-level issues in Elixir and Erlang applications, reinforcing New Relic's commitment to performance and resilience.