May 2022 Summaries

New Relic is a monitoring and observability solution that helps sysadmins and developers understand their infrastructure systems by providing real-time data and insights. It supports various operating systems, including Windows, Linux, macOS, Docker, and Kubernetes, and can be installed on any system, including virtual machines, private or public clouds, and even local machines. The tool provides a range of features, such as dashboards, alerts, metrics, logs, and event explorers, to help identify root cause issues in infrastructure systems. Installation is relatively straightforward, requiring only an active New Relic account, a system to instrument, and internet connectivity. Once installed, users can use stress testing tools like stress-ng to simulate various scenarios and test their system's performance in real-time.

Leon Adato, a developer relations advocate at New Relic, discusses the process and benefits of using New Relic's infrastructure monitoring tool to gain insights into various system metrics such as CPU, RAM, storage, and network traffic. Drawing from his extensive IT background, Adato explains how New Relic allows sysadmins to set up monitoring with ease across different platforms, including Windows, Linux, macOS, and cloud environments, emphasizing the importance of non-production testing. He highlights the tool's dashboards and alert features, which help automate issue detection and resolution, thereby reducing reliance on user-reported problems. Adato also outlines the installation process of New Relic, which is user-friendly and requires minimal technical setup, and encourages experimentation with stress-testing tools to simulate system load and observe the results in New Relic's dashboards. The blog reflects Adato's personal experience and expertise in monitoring and observability, offering practical insights while noting the importance of environment-specific solutions.

The recent acceleration of digital transformation has increased the workload on technology teams, exacerbating the shortage of skilled resources. To mitigate this shortage, automation and autonomous self-service are crucial. Automating observability can significantly decrease the workload of SREs and tech teams. New Relic provides various ways to automate different layers of the observability stack, including telemetry collection, visualization, alerts, platform governance, and custom applications. The guided installer, APM and infrastructure agents, OpenTelemetry SDKs, MELT APIs, entities, workloads, dashboards, and custom applications can be used to automate these processes. Additionally, New Relic offers free accounts with 100 GB/month of free data ingest, allowing users to try out its features and start automating their observability stack.

As organizations face increased workloads due to accelerated digital transformation, automating observability processes has become essential for technology teams, particularly those utilizing New Relic. The blog post outlines strategies for automating various layers of the observability stack, including telemetry collection, visualization, alerts, and platform governance. New Relic facilitates this automation through tools like guided installers, APM and infrastructure agents, OpenTelemetry SDKs, and MELT APIs, which help streamline data collection and integration. Visualization is enhanced by entities and workloads that dynamically display relevant metrics, while customizable dashboards and applications allow teams to tailor their observability views. The post emphasizes setting up actionable alerts and recommends automating user management through identity providers for efficient platform governance. By utilizing these tools and practices, teams can reduce workloads and improve system visibility, with New Relic providing open-source solutions and community support to aid in this transition.

To truly understand how your K8s cluster impacts your apps, you need to analyze and correlate performance data from both application-level metrics and infrastructure signals. New Relic provides a seamless developer experience by correlating OpenTelemetry (OTel) application-level metrics with Kubernetes infrastructure metrics, making it easier to view the entire landscape of telemetry data and work across teams to get faster Mean Time to Resolution (MTTR) for issues in your Kubernetes environment. To set up this integration, you need to install New Relic's Kubernetes integration in your cluster, instrument your applications with OpenTelemetry, and deploy the OpenTelemetry collector as an agent on every node in your cluster. The collector can then receive telemetry data from your applications and enrich it with metadata from your infrastructure, allowing for a single UI to view both OTel and Kubernetes performance data.

Kubernetes' integration with New Relic offers developers a unified platform to monitor and correlate application and infrastructure performance metrics, enhancing their ability to resolve issues efficiently. By using OpenTelemetry (OTel) to capture application-level data and integrating it with Kubernetes infrastructure metrics, New Relic provides a comprehensive view of telemetry data. This integration involves setting up the OTel collector with environment-specific metadata, allowing developers to visualize their OTel and Kubernetes data in a single interface. Additionally, the OpenTelemetry collector can be deployed as an agent across the cluster, enriching telemetry data with infrastructure-specific information. This approach helps reduce Mean Time to Resolution (MTTR) by enabling quick identification of errors and performance issues, while also offering seamless navigation between application and infrastructure monitoring within New Relic's interface.

Instrumenting your Crystal application with OpenTelemetry and New Relic can help you find and fix errors and performance issues in your production systems. The process involves adding instrumentation to your code, configuring OpenTelemetry, and sending data to New Relic using the OTLP/HTTP protocol. With auto-instrumentation, you can collect data on how long it takes to calculate Fibonacci numbers in your application, as well as what numbers are being calculated, without manually adding spans like this in a production environment. The example code demonstrates how to instrument a simple Crystal service and send its traces to New Relic for visualization.

The FutureStack 2022 conference featured a keynote address by Bill Staples, CEO of New Relic, which covered new platform capabilities, product demos, and partnership announcements with Microsoft Azure, as well as vulnerability management and security solutions. The event also included live demos, such as CodeStream, a tool that supercharges development workflows, and the launch of a $100,000 Impact Fund to support nonprofit technology projects. Additionally, New Relic announced partnerships with Platzi, Learn with Jason, Wilco, and other organizations, including scholarships for students and a gamified programming experience called League of Observability. The conference also featured inspiring talks from Dr. Moogega Cooper, NASA's planetary protection engineer, and Steve Wozniak, Apple co-founder, who discussed the importance of taking fearless risks and pursuing one's passion.

Crystal is an object-oriented programming language influenced by Ruby, with some features from Go and other languages, which is increasingly being used by enterprise companies for core systems. Despite its growing use, observability platforms have not extensively integrated with Crystal, creating a need for solutions to monitor and troubleshoot these systems. The article explores how to instrument Crystal applications using New Relic and OpenTelemetry, providing a detailed guide on configuring these tools to monitor the performance of applications, such as a basic Fibonacci number calculator service. It explains how to set up OpenTelemetry to send data to New Relic, emphasizing the importance of tracing and logging to capture detailed metadata about application processes. The piece also touches on the ongoing development of Crystal's OpenTelemetry support and encourages users to explore further resources for deeper understanding and implementation of observability strategies.

FutureStack 2022 featured a series of significant announcements and partnerships aimed at enhancing New Relic's platform capabilities and user experience. The event began with a keynote by New Relic CEO Bill Staples, who highlighted new product capabilities and collaborations, such as the partnership with Microsoft Azure, allowing New Relic to be integrated as a default observability platform within Azure. The conference also introduced Security RX, a tool for aggregating vulnerability data, and celebrated the achievements of participants in FutureHack 2022. Dr. Moogega Cooper and Apple co-founder Steve Wozniak delivered inspiring speeches on innovation and technology, while New Relic announced a $100,000 Impact Fund to support nonprofit technology projects and educational partnerships with platforms like Platzi and Wilco. These initiatives underscore New Relic's commitment to advancing digital transformation and observability in the tech community.

The O11y Awards recognize companies that demonstrate excellence in observability, customer focus, business impact, and innovation using New Relic's platform. Companies like Mercado Bitcoin, Credit Karma, BEES, EverQuote, and American Red Cross have been recognized for their outstanding achievements in observability, such as driving exponential growth, establishing an observability practice, or demonstrating a strong commitment to serving customers. Individuals like Mark Huber, Zainul Masadi, Peter Espe, and Shilpa Patil have also received awards for their contributions to observability using New Relic at their companies, including championing the platform's adoption, building dashboards and integrations, and promoting best practices. The awards aim to highlight innovative use cases of New Relic's platform and inspire others to adopt similar approaches.

New Relic's O11y Awards recognize companies and individuals demonstrating excellence in observability, highlighting how they leverage New Relic's platform to drive growth, innovation, and customer satisfaction. Mercado Bitcoin, Credit Karma, BEES, EverQuote, and the American Red Cross were acknowledged for their innovative use of observability to enhance operational efficiency, customer experience, and digital transformation. The awards also celebrated individual contributions from Mark Huber at Cox Automotive, Zainul Masadi at Tokopedia, Peter Espe at Credit Karma, and Shilpa Patil at AmerisourceBergen for their roles in advancing observability practices within their organizations. These efforts reflect the growing importance of comprehensive observability strategies in today's digital landscape.

New Relic has announced several capabilities that make it easier for teams to monitor, debug, and improve their entire stack-and-embed observability practices and telemetry data throughout the entire software development lifecycle. These include Infinite Tracing, regional services, external service monitoring, machine learning model monitoring, infrastructure and network monitoring, Kubernetes with Pixie, log management, Real User Monitoring (RUM) and synthetic monitoring, errors inbox, developer experience, service level management, customizable navigation, omnimap workload views, alerts and AIOps, security, pricing, integrations and Instant Observability. Additionally, New Relic has introduced a new data onboarding experience, enhanced NRQL querying, native OTLP endpoint, public issue tracker, Platzi student partnership, and several other features to improve the user experience and extend the value of observability.

At FutureStack 2022, New Relic unveiled over 30 new capabilities designed to enhance observability across software development lifecycles, integrating telemetry data and improving monitoring, debugging, and overall stack performance. Key announcements included the expansion of Infinite Tracing to support browser, mobile, and AWS Lambda traces, along with regional services in the EU and Asia-Pacific regions. New features for application performance monitoring, machine learning model monitoring, and enhanced Kubernetes integration were also introduced, providing more efficient diagnostics and observability in distributed systems. The platform now offers advanced infrastructure and network monitoring capabilities, enriched anomaly detection, and improved alert accuracy, alongside new security features under Security RX and pricing options with Data Plus. Integration with Microsoft Azure and a partnership with Platzi further extend New Relic's reach into cloud services and educational realms, respectively, as the company continues to focus on providing seamless and powerful observability solutions.

The updated Kubernetes integration (v3) significantly reduces CPU and memory usage, adds support for external control planes like Rancher, and improves log messages to help identify issues. The new integration provides a 80% reduction in memory footprint, improved troubleshooting with enhanced logs and process cycle, more configurable components, and flexible scraping intervals. To upgrade the integration, users can run a single command using Helm, and no configuration updates are required. The new integration follows Kubernetes best practices and includes improvements such as decoupled component configuration, reduced complexity in the cache system, and support for complex data structures. The updated architecture allows for smarter decision-making at scheduling and deployment time, ensuring that issues are detected more easily.

As Kubernetes environments grow more complex with increased scaling, New Relic's updated Kubernetes integration (v3) aims to simplify monitoring by enhancing performance and configurability while reducing resource usage. This update includes a significant reduction in CPU and memory consumption, support for external control planes like Rancher, and improved log messages for easier troubleshooting. The integration now allows for a more flexible configuration with the use of YAML files instead of environment variables, enabling support for complex data structures. It also introduces a sidecar pattern for better error handling and lifecycle management, and allows manual configuration of scraping intervals to suit specific needs. The upgrade process is streamlined with a compatibility layer that maps old values to the new system, minimizing breaking changes. Throughout development, rigorous testing ensured a smooth transition from v2 to v3, addressing issues and optimizing the user experience.

New Relic has undergone significant evolution over the years, shifting from an all-in-one observability platform to a consumption-based business model and rebranding itself as a company that prioritizes customer success. The new brand reflects this shift, with a modern look and feel, voice and tone, tagline, logo, colors, and website that focus on empowering software engineers to do their best work every day using data-driven insights. The new tagline "Data for Engineers" emphasizes the company's commitment to providing engineers with access to all their telemetry in a single unified platform, paired with powerful full-stack analysis tools, to help them overcome opinions and fill gaps in their workflow.

New Relic has undergone significant transformations, evolving from a traditional brand to a modern, customer-focused observability platform with a new brand identity launched at FutureStack 2022. This rebranding reflects their commitment to supporting software engineers by providing a unified data platform that consolidates metrics, events, logs, and traces, enhancing their ability to plan, build, deploy, and manage software effectively. The new visual identity includes a logo that symbolizes the collaboration between development, security, and operations teams through a hexagon formed by brackets, and a dominant green color representing optimal system performance. New Relic's shift to a consumption-based business model and usage-based pricing aligns their entire strategy around customer success, aiming to empower engineers with data-driven insights to improve digital experiences.

Apache Kafka is an open-source event streaming platform used by thousands of companies, including New Relic. It's distributed, highly scalable, and fault-tolerant but can be challenging to monitor due to its decoupled producer-consumer architecture and asynchronous processes. Distributed tracing with OpenTelemetry can help monitor Kafka clusters by providing context to messages, allowing for the analysis of bottlenecks, downstream impacts, data loss, consumer polling, and fine-tuning of env variables for producers. The code example demonstrates how to inject context into messages from a producer using OTel Propagators API and how to enable consumers to read incoming message context. Visualizing the data in Jaeger or an observability platform like New Relic provides valuable insights into performance bottlenecks and optimization opportunities.

Apache Kafka is a robust, open-source event streaming platform used by numerous companies for handling real-time data, yet monitoring its clusters can be challenging due to its distributed nature and asynchronous processes. Distributed tracing, particularly through OpenTelemetry, provides a solution by enabling the tracking of requests across systems, thereby improving visibility into Kafka's operations and allowing for the swift identification of bottlenecks and optimization of data pipelines. New Relic, a full-stack observability platform, utilizes Kafka to process vast amounts of data and applies distributed tracing to enhance its telemetry data platform, particularly by optimizing "time to glass," which measures the speed at which data becomes queryable. Implementing distributed tracing involves adding context to Kafka message headers, using tools like the otelsarama library, and visualizing data with platforms like Jaeger or New Relic. This setup aids in optimizing Kafka configurations and tracking potential data loss, thereby enhancing overall system performance and reliability.

New Relic provides tools to help developers prioritize and resolve errors, allowing them to focus on writing new code rather than battling technical debt. The "Errors Inbox" feature offers a unified view of all errors across multiple sources, enabling teams to organize their backlog more effectively and achieve data-driven sprint planning sessions. By applying tags to environment resources, teams can create workloads that group entities together for easier management. Once errors are identified, they can be assigned to engineers or teams using the Errors Inbox UI, which also integrates with JIRA for seamless issue creation. Additionally, the CodeStream extension allows developers to quickly troubleshoot critical errors directly within their code editor, reducing context switching and improving productivity.

Developers often face the challenge of managing incidents and technical debt, which detracts from their ability to focus on creating new features. New Relic offers solutions to streamline this process, enabling developers to prioritize and resolve errors efficiently, thereby improving system reliability with each sprint cycle. Utilizing tools like errors inbox, workloads, CodeStream, and Jira integration, developers can organize, triage, and assign errors effectively. Errors inbox provides a unified view of errors from various sources, allowing for improved sprint planning and backlog management. Workloads help group and monitor entities based on tags, facilitating error prioritization. The integration with CodeStream allows developers to transition seamlessly from error identification to code resolution within their IDEs, reducing context switching. New Relic's tools aim to enhance developers' productivity by minimizing time spent on error management and maximizing focus on innovative development.

When system performance suffers, you need to know if it's due to your code, your infrastructure, or the underlying network. With New Relic's network monitoring capabilities, you can see how network performance and overall system performance impact each other, without needing extensive knowledge of networks and their protocols. A new onboarding experience, helpful landing pages, and curated UIs have been created to make it easy for non-network-savvy engineers and those new to observability to understand network monitoring telemetry right away. Guided prescriptive onboarding experiences, purpose-built landing pages, and curated visualizations help identify the most important network-related use cases for your business, making it easier to analyze data. The new capabilities allow you to set up New Relic network integrations step-by-step, deploy network monitoring in as little as five minutes, configure alerts, set up workloads, and understand network performance with curated user interfaces. With these new features, you can quickly grasp changes in your network telemetry, navigate to differences, and determine if issues are related to the network or not.

New Relic introduces enhanced network monitoring capabilities designed to help users quickly identify performance issues by offering guided onboarding, purpose-built landing pages, and curated user interfaces. These tools allow users, regardless of their network expertise, to understand and analyze network telemetry and its impact on overall system performance. The new onboarding experience simplifies the setup process, enabling users to configure network integrations with a few commands, while landing pages provide a high-level overview of network performance and the ability to configure alerts. Curated UIs highlight relevant metrics without requiring custom dashboard creation, and additional features like deviating attributes from Lookout help users pinpoint changes in network telemetry. Users are encouraged to explore these features through New Relic's platform, which offers a free account with data ingest and user access.

To enhance error visibility and filtering, New Relic has introduced OpenTelemetry-based tracing span data in its Errors inbox feature, allowing users to triage errors more effectively with detailed view of stack traces, span events, and attributes. This innovation is unique to the New Relic platform, providing a user-friendly experience for instrumented services. The feature can be easily accessed by navigating to the service entity in the Services - OpenTelemetry view, without requiring any workloads setup. With this update, users can now view errors at both the entity and workload level, spot patterns, and pinpoint issues, making it easier to troubleshoot performance and behavior of applications. Additionally, New Relic supports OpenTelemetry telemetry data for various observability capabilities such as trace maps, AutoMaps, and Related Entities, providing a comprehensive view of system behavior. The feature offers enhanced error visibility, filtering, and aggregation features, making it easier to detect, prioritize, and triage errors.

New Relic has introduced OpenTelemetry-based tracing span data into its Errors Inbox, enhancing error triage capabilities for services instrumented with OpenTelemetry tracing. This update allows users to view tracing details such as stack traces, span events, and span attribute data directly within the Errors Inbox, facilitating the identification and resolution of errors. OpenTelemetry, a set of tools for collecting and exporting metric, log, and trace data, is supported by New Relic, which is a contributor to the Cloud Native Computing Foundation's OpenTelemetry project. The integration provides improved error visibility and filtering at both the entity and workload levels, making it easier to detect patterns and address issues. Additionally, New Relic offers integrations with platforms like Slack, Jira, and CodeStream to streamline error management and collaboration. The Errors Inbox is available for both core and full platform New Relic users, and resources such as a demo video, documentation, and a setup guide are provided to help users leverage these new capabilities.

If you're joining us at FutureStack 2022 in Las Vegas from May 17-19, you can explore three tracks to make programming easier: Operate for prod observability knowledge, Shift Left for pre-prod practices, and Expand for scaling across teams. You can also attend hands-on labs, events, and sessions led by experts like Phil Weber and Ryan Cheng, covering topics such as true observability, dashboard design, monitoring Kafka without instrumentation, and distributed tracing. The conference features a diverse lineup of speakers from companies like IBM, Chegg, and Lacework, providing a platform for software engineering professionals to learn and network.

FutureStack 2022, held at the Cosmopolitan Hotel in Las Vegas from May 17-19, offers a comprehensive lineup of sessions designed to enhance observability knowledge across three main tracks: Operate, Shift Left, and Expand. Attendees can engage in hands-on labs and technical deep dives, exploring topics such as full-stack monitoring, AIOps, MLOps, and in-context logging, with opportunities to hear from industry leaders and participate in events like the FutureHack hackathon. The conference features notable sessions, including "5 key steps to true observability" by Phil Weber, "Monitoring Kafka without instrumentation using Pixie" by Ryan Cheng and Anton Rodriguez, and "Troubleshoot faster with distributed tracing" by Liam Hurrell. Highlighting contributions from companies like IBM, Chegg, and Lacework, FutureStack aims to provide insights into observability practices that improve uptime, reliability, and operational efficiency.

As a leading enterprise video platform in Europe, movingimage was facing issues with high availability and performance in its video content management system, which was running on oversized virtual machines and overprovisioned containers in Microsoft Azure. The company's engineering team was struggling to diagnose and fix problems due to the lack of reliable monitoring tools. To address this, they deployed an observability platform using New Relic, which provided visibility down to the code level with application metrics that informed data-driven decisions. By creating a container dashboard, they were able to identify wasted resources and optimize Kubernetes container sizes, cutting costs in half and improving stability and performance. The company's efforts have led to faster incident response and troubleshooting, resulting in customers noticing significant improvements.

Movingimage, a leading European enterprise video platform, faced challenges with overprovisioned Kubernetes containers and oversized virtual machines after migrating to Microsoft Azure, which led to inefficient resource usage and stability issues. The company lacked the tools to accurately assess container resource requirements, leading to manual log reviews. With the introduction of New Relic as an observability platform, the DevSecOps team gained detailed insights into resource consumption, enabling them to make data-driven decisions and optimize container sizes. This optimization significantly reduced the number of nodes in production, cutting compute costs by about 60% and improving stability and performance, garnering positive feedback from customers. The initiative, which involved creating a custom dashboard to guide resource allocation and provide recommendations for rightsizing, reflects movingimage's commitment to becoming a more data-driven organization.

This five-part series explores automating common DevOps tasks in New Relic using Python and the NerdGraph API. The series covers auditing services, automating tags, exporting telemetry data, exporting dashboards, and managing license keys. Each part provides code examples and explanations of how to use the NerdGraph API to perform these tasks. The series aims to help users automate common DevOps tasks and improve their observability practice. New Relic offers a free account that includes 100 GB/month of free data ingest, one free full-access user, and unlimited free basic users. The views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic.

In the final installment of a five-part series on automating DevOps tasks using New Relic, the focus is on managing license keys through the New Relic Query Language (NRQL) and Python. The series covers creating, reading, updating, and deleting license keys using the NerdGraph API, offering sample code for each task. Regular rotation of license keys is emphasized for security, especially in industries adhering to standards like PCI DSS. The post includes Python methods for interacting with the NerdGraph API, demonstrating how to execute GraphQL queries for various license key operations. The author, Steve Ng, a Senior Manager of Developer Relations at New Relic, highlights the importance of integrating DevOps into observability practices and encourages readers to explore further resources provided by New Relic, including a free account offering and various integrations.