May 2024 Summaries

New Relic runs a large streaming platform that processes several petabytes of observability data daily, relying heavily on Apache Kafka for efficient data processing and low latency. To maximize efficiency and reliability, it's crucial to tune the Consumers, which receive data from the stream processing applications. High-quality instrumentation is necessary to diagnose performance and resource usage, and New Relic provides a curated UI to collect and analyze metrics. Understanding how to interpret these metrics is key to detecting and remedying common issues such as leaving data on the table or not waiting for more data. By tuning settings like fetch.max.bytes and fetch.min.bytes, and considering factors like compression rate and memory use, developers can optimize their Kafka Consumers for better performance and cost savings. Additionally, New Relic is exploring ways to provide dynamic and automatic tuning for Kafka Consumers with simpler constraints, while also offering alerting tools to help identify wasting cluster or compute resources.

New Relic operates a large streaming platform using Apache Kafka, processing petabytes of observability data daily. Key to optimizing Kafka's performance is the effective tuning of Kafka Consumers, which receive data from the stream. The article emphasizes the importance of high-quality instrumentation to monitor and adjust Kafka Consumer metrics for better efficiency and cost management. It provides insights into configuring fetch size and latency settings such as fetch.max.bytes, max.partition.fetch.bytes, fetch.min.bytes, and fetch.max.wait.ms to avoid inefficiencies. The New Relic Kafka UI aids in visualizing these metrics, enabling users to identify and address issues like hitting fetch size limits or inefficient data batching. While the article focuses on the Apache Kafka Java Client, the principles are applicable to other language clients. It also mentions potential future developments for automated tuning solutions and encourages using alerts to detect resource wastage.

New Relic is a critical tool for developers and executives alike, offering real-time visibility and insights into system performance, user experience, and customer journeys. The platform provides key performance indicators (KPIs) in easy-to-understand dashboards, allowing executives to quickly drive productivity, break down silos, communicate with business leaders, deliver proactive support for peak demand performance, and simplify security to reduce application risks. Through its free tier and robust features, New Relic enables companies to identify and address persistent challenges, improve system resilience, and make informed decisions with data-driven insights.

Observability is highlighted as a crucial tool for both developers and C-level executives, providing essential real-time insights that enhance productivity and customer experience. New Relic, a platform offering these observability solutions, is praised for its ability to quickly deliver value through its free tier, which helps companies identify and address challenges without budget constraints. Executives from various companies, including Statista, M&S, and 10x Banking, emphasize New Relic's role in breaking down silos, improving system resilience, and enabling informed decision-making across teams. The platform supports effective communication by translating technical metrics into business-relevant insights, which is especially beneficial during peak demand periods such as holiday sales or major sports events. Furthermore, New Relic aids in simplifying security processes, crucial for companies under strict privacy regulations, by surfacing vulnerabilities to developers, thus facilitating proactive risk management.

Observability is a crucial aspect of modern software systems that enables businesses to gain insights into their operations and make data-driven decisions to drive success. It encompasses three pillars: logs, metrics, and traces, which provide a comprehensive picture of a system's performance. By monitoring key metrics such as latency, traffic, error rates, and saturation, companies can ensure their systems are operational and optimized for the best user experience. Observability also helps in mitigating risks, making informed decisions, and driving business outcomes through improved customer satisfaction, operational efficiency, and strategic decision-making. To truly harness the power of observability metrics, teams should adopt best practices such as comprehensive coverage, real-time monitoring and alerts, contextual analysis, collaborative review, and leveraging AI and machine learning. By doing so, companies can maximize the value of observability and drive better business results.

Observability extends beyond maintaining system uptime to providing insights that enhance business performance, utilizing the three pillars of logs, metrics, and traces to assess the health and effectiveness of software systems. Logs offer chronological narratives essential for debugging, metrics quantify system performance influencing data-driven decisions, and traces reveal the flow of requests to identify bottlenecks, thereby linking technical metrics to business outcomes. Key metrics like latency, traffic, error rates, and saturation are vital for informed decision-making that improves customer experience, operational efficiency, and risk mitigation. Best practices such as comprehensive coverage, real-time monitoring, contextual analysis, and collaborative reviews, along with AI and ML tools, empower teams to optimize performance and prevent potential issues. Companies like D24 leverage observability to maintain high service-level agreements and drive innovation, demonstrating its crucial role in aligning technical performance with business success.

Observability, crucial for modern software systems, is explored as a vital tool for understanding and enhancing business performance by providing insights into system health and effectiveness. The concept is built on three pillars: logs, metrics, and traces, each contributing uniquely to a comprehensive understanding of a system's state. Logs detail chronological events, metrics offer numerical performance data, and traces illustrate request flows, all essential for debugging and decision-making. Observability metrics such as latency, traffic, error rates, and saturation are emphasized for their role in improving user experience, operational efficiency, risk mitigation, and data-driven decision-making. The blog highlights the importance of aligning these metrics with business objectives to foster collaboration between engineering and business teams, leveraging tools like AI and machine learning for predictive analytics. Companies such as D24 and Statista exemplify the successful integration of observability metrics to drive growth and innovation, demonstrating the transformative potential of these tools in achieving business success.

Observing WebAssembly applications presents unique challenges due to its design and execution environment. The Bytecode Alliance promotes standards and tools for enhancing security and usability, including better support for observability. Blazor WebAssembly allows developers to build interactive client-side UIs using .NET rather than JavaScript, leveraging full-stack capabilities of .NET on both server and client sides. OpenTelemetry can be integrated with .NET to provide comprehensive observability for these applications, but manual instrumentation is required for detailed visibility into backend operations. The project demonstrates how to instrument the backend with automated OpenTelemetry instrumentation and export telemetry data to an OpenTelemetry backend, such as New Relic. For the frontend, real user monitoring capabilities are available through the New Relic browser API, allowing developers to observe high-level telemetry from the frontend.

Observing WebAssembly applications poses unique challenges due to its binary execution in a browser sandbox, which complicates direct introspection compared to traditional web applications. The Bytecode Alliance is working on enhancing WebAssembly's security and observability standards, and Blazor WebAssembly, a framework that compiles C# into WebAssembly, enables developers to use .NET for both client-side and server-side development, providing a streamlined and high-performance development process. The integration of OpenTelemetry with Blazor WebAssembly allows for comprehensive observability by collecting and exporting telemetry data, though the lack of direct HTTP request capabilities in WebAssembly complicates frontend instrumentation. Automated and manual instrumentation using OpenTelemetry in the backend can capture detailed telemetry data, while frontend monitoring can be achieved through real user monitoring capabilities, such as those provided by New Relic. Despite these challenges, the article illustrates a practical approach to gaining insights into Blazor WebAssembly applications by leveraging existing tools and platforms, with the expectation that ongoing community efforts will continue to simplify observability for these applications.

The American Academy of Pediatrics (AAP) recently completed a rebundling project to standardize their approaches with New Relic's observability mindset, focusing on workloads as a key tool for collaboration and ease of working across the organization. By creating an aggregated view of the health and activity of specific business units or teams, workload monitoring enables teams to see everything that affects AAP at once, making it easier to support and maintain services. The introduction of alerts and threshold checks has improved issue detection and response times, allowing the support team to provide more accurate and timely responses to users. Workloads also facilitates communication across teams and departments, ensuring that upper management and customer service are informed about issues and their resolution. By leveraging New Relic's workload monitoring capabilities, AAP has been able to streamline its operations, improve performance, and reduce downtime, ultimately enhancing the user experience for its members and customers.

The American Academy of Pediatrics (AAP) has successfully completed a project to consolidate their various online services and sites into a single platform under the aap.org website, aiming to streamline operations and enhance observability. This transition to a unified system has allowed for the adoption of standardized practices using New Relic's observability tools, notably the 'workloads' feature, which provides a comprehensive view of system health and activity, thus improving issue detection and resolution. The rebundling efforts have facilitated better collaboration among teams and allowed for more efficient monitoring and management of services, reducing the complexity that previously hampered growth. By implementing a train-the-trainer approach, the AAP has enhanced its staff's ability to understand and utilize observability metrics proactively. Additionally, by fine-tuning alert systems and employing synthetic monitoring, the organization has improved performance and user experience, while maintaining effective communication with internal and external stakeholders about system status and issues. The use of New Relic's tools has significantly improved the AAP's operational efficiency by providing greater insight and control over their infrastructure.

The mission at New Relic is to empower engineers with a data-driven approach to planning and building great software. To achieve this, they aim to set the industry standard for observable application security with the introduction of the Secure Developer Alliance. This alliance brings together industry leaders such as Gigamon, Lacework, FOSSA, Aviatrix, and Opus to provide customers with a best-of-platform approach and pragmatic research, education, and guidance to implementing observable application security. The goal is to ensure that observable application security delivers value to customers through best-of-breed tools, research, education, and guidance, reducing build time and runtime application risks while maintaining developer productivity and software release velocity.

A new approach to vulnerability management is being introduced that prioritizes vulnerabilities based on a weighted system considering severity, exploitability, and known ransomware associations. This shift aims to reduce alert fatigue and provide developers with actionable intelligence to focus their efforts on the most critical security issues. By integrating Exploit Prediction Scoring System (EPSS) forecasts, active exploit data, and interactive application security testing (IAST), the system helps teams identify vulnerabilities that pose a tangible risk to their applications. The weighted system provides an objective measure by considering four key elements: CVE severity, EPSS, known ransomware associations, and IAST exploitable vulnerability. This approach enables laser-focused resource allocation, proactive defense, and adaptive security, ultimately empowering organizations to shore up their defenses and shape a more secure future for the digital world.

Security vulnerability management in software development is evolving from a severity-based approach to a more nuanced, developer-centric strategy that prioritizes vulnerabilities based on actual risk and exploitability. This shift addresses the limitations of traditional methods, such as the Common Vulnerability Scoring System (CVSS), which may not accurately reflect the threat a vulnerability poses. By incorporating data from the Exploit Prediction Scoring System (EPSS), real-time testing methods like Interactive Application Security Testing (IAST), and information on active malware campaigns, organizations can focus their resources on vulnerabilities that pose a tangible risk. This refined prioritization method aims to reduce alert fatigue and improve the security posture of applications, aligning efforts with the most pressing threats and enabling a more proactive defense. New Relic's Security RX platform supports this approach, integrating various data sources to enhance security teams' ability to protect digital assets effectively.

New Relic is enhancing its commitment to empowering engineers with a data-driven approach by introducing the Secure Developer Alliance, a coalition with industry leaders such as Gigamon, Lacework, FOSSA, Aviatrix, and Opus. This initiative aims to set a new industry standard for observable application security by providing a best-of-platform approach alongside pragmatic research, education, and guidance to improve security outcomes without hindering developer productivity. Observable application security at New Relic leverages full-stack observability data and security telemetry to minimize application risks while maintaining software release velocity. The Secure Developer Alliance addresses the need for effective security tools that do not overburden developers with unnecessary alerts, allowing businesses to innovate and grow competitively. The program supports DevOps in managing security and customer experience, aligning with the dual objectives of enhancing productivity and driving business innovation.