July 2024 Summaries

The development and deployment of AI systems are subject to growing regulatory scrutiny, with individual states in the US and the European Union enacting laws and regulations to protect citizens' rights, privacy, and security. The EU's Artificial Intelligence Act, which will come into force on August 1, 2024, places significant obligations on developers, deployers, and providers of high-risk AI systems, including requirements for logging capabilities, record-keeping, and disclosure of potential harms. New Relic, a company that provides observability solutions, highlights its ability to help companies meet these compliance needs, particularly in terms of logging requirements under Article 12 of the EU AI Act, through its support for various log-forwarding options, including Fluent Bit, Fluentd, Logstash, and cloud service provider logs. The company's platform can also provide comprehensive observability solutions for monitoring and logging needs, making it a valuable tool for companies navigating the complex landscape of AI regulations.

In the evolving landscape of artificial intelligence (AI), legislative bodies in the United States and the European Union are implementing regulations to ensure the responsible development and deployment of AI systems, particularly those deemed high-risk. While individual U.S. states like Colorado and Utah have initiated their own AI laws, the federal government, under President Biden's executive order, aims to establish guidelines through agencies like the National Institute of Standards and Technology. The European Union's forthcoming Artificial Intelligence Act, set to be enforced in August 2024, presents a broader regulatory framework, mandating logging and monitoring capabilities for high-risk AI systems under Article 12. Platforms like New Relic offer solutions to help companies meet these compliance challenges by providing comprehensive observability tools that facilitate compliance with logging requirements, enabling businesses to maintain transparency and accountability in their AI operations. As AI technologies advance, businesses are encouraged to leverage tools like New Relic to navigate the complexities of compliance and maintain oversight over their AI systems.

To help answer questions about end-to-end tests, teams can collect and use test data by sending custom events to New Relic via its event API. By tracking code coverage, flakiness, and performance, teams can improve their testing strategies, empower data-driven decision-making, and enhance the overall quality of their software delivery lifecycle. With features like dashboards, NRQL queries, and observability practices, New Relic provides a platform for teams to visualize test metrics, identify potential gaps in code coverage, and make informed decisions about increasing investment in testing. By following best practices and avoiding common pitfalls, teams can create custom dashboards that foster empowerment and collaboration, ultimately leading to fewer incidents, reduced risk, faster lead times, and a possible reduction in the rate of change failures.

New Relic's blog post explores the integration of data into testing processes to address challenges such as long execution times and test coverage tracking. It outlines the use of New Relic's event API for sending custom events related to testing, which aids in gathering detailed insights into code coverage and test performance. The post emphasizes the importance of tracking metrics like code coverage, test flakiness, and test duration to refine testing strategies and improve performance. It highlights the synergy between testing, observability, and DevOps, referred to as TOAD, which enhances decision-making and quality assurance. Additionally, the post encourages the use of centralized dashboards for data visualization and governance, enabling teams to collaborate effectively and maintain quality standards across projects.

Next.js 13 has adopted the App Router as its recommended way to handle routing, leveraging React's latest features for optimized speed and performance. To set up application performance monitoring with New Relic, install the agent using npm, modify the next.config.js file, and add configuration files such as newrelic.js and .env. The integration provides comprehensive observability of server-side activities, while also allowing client-side telemetry through the injection of the New Relic browser agent. To capture detailed error information, handle both client-side and server-side errors using code samples provided in the blog post, which can be found in the newrelic-node-examples GitHub repository.

Next.js 13 introduces the App Router, which utilizes React's latest features like Server Components and Streaming to enhance web application efficiency. This blog post guides users on setting up application performance monitoring for both server-side and frontend aspects using the App Router, offering full-stack observability in Next.js applications. It details the integration of New Relic for monitoring, starting with the installation of the New Relic Node.js agent, which now includes the @newrelic/next package by default, enabling server-side monitoring without additional installations. Users are instructed to modify the next.config.js file and package.json scripts to ensure proper instrumentation. The post also covers injecting the New Relic browser agent for client-side telemetry and demonstrates error tracking through example code for both client-side and server-side errors. The blog concludes by directing users to additional resources, including GitHub repositories and the Explorers Hub for community support, while emphasizing the authors' expertise in JavaScript and data monitoring.

A recent platform update by Crowdstrike caused widespread outages across multiple industries, including airlines, emergency call centers, and hospitals, highlighting the critical importance of observability in dealing with system outages. Observability tools like New Relic provide real-time insights into system performance and health, enabling businesses to quickly identify dependencies and take corrective action. As digitalization continues to advance and AI becomes increasingly integrated into daily life, businesses must prioritize observability to ensure the reliability of their digital experiences, as even seemingly well-monitored systems can still miss critical issues without the right tools.

On July 19, 2024, a Crowdstrike platform update led to widespread outages in Windows-based systems, affecting critical services like airlines, hospitals, and banks, and highlighting the importance of observability in managing such incidents. Observability tools, such as New Relic, provide real-time insights into system performance and interdependencies, enabling swift identification and resolution of issues. As the digital world becomes increasingly intertwined with everyday life and as AI integration deepens, the need for robust monitoring solutions becomes more critical. Observability not only aids in fixing immediate problems but also in understanding system dynamics, ensuring businesses can quickly recover from disruptions. This incident underscores the growing complexity of outages in a digitally dependent world and the vital role of observability in maintaining reliable digital operations.

In today's rapidly evolving landscape of cloud-based and virtual environments, developers are seeking innovative ways to ensure the reliability and security of their applications. Traditional security reference architectures have relied on methods like SIEM and log analysis tools, but a new player is emerging - application performance monitoring (APM) data. APM data offers a unique approach to early security warnings, but collaboration between application and security teams is crucial to bridge the gap. APM tools hold a treasure trove of data that can benefit security professionals, providing visibility into external resource access, database query monitoring, anomaly detection, performance monitoring, and more. The integration of APM and application security (AppSec) not only optimizes application performance but also enhances security by detecting and preventing various types of threats, including SQL injections, cross-site scripting, denial-of-service attacks, malware infections, unauthorized access, data exfiltration, and code vulnerabilities. As APM tools continue to evolve and integrate with security practices, both developers and security teams will have a powerful ally in ensuring the performance, reliability, and security of their applications.

In the evolving landscape of cloud-based and virtual environments, the integration of Application Performance Monitoring (APM) data with security measures is reshaping application development and deployment. Traditional security architectures often lack access to APM data, which provides unique insights into application performance and potential vulnerabilities. By collaborating, application and security teams can leverage APM tools to detect anomalies such as SQL injections, cross-site scripting, denial-of-service attacks, and unauthorized access, thereby enhancing security protocols. New Relic's APM platform exemplifies this integration by offering features like AI-powered prioritization, interactive application security testing, and data-driven workflows that boost productivity and security efficiency. This synergy between APM and application security not only optimizes performance but also positions APM as a crucial part of modern defense strategies, promising a future where performance, reliability, and security are seamlessly intertwined.

The Water Project, a nonprofit organization, leverages an observability platform to manage its tech stack efficiently with limited resources. They focus on two key goals: bug fixing and performance monitoring. By using New Relic's platform, they can monitor their tech infrastructure, track bugs and errors, and optimize server performance, which has resulted in improved user experience, faster response times, and reduced downtime. The organization also uses the platform to automate processes, squash errors, and maintain a high level of quality, ultimately driving equitable access to technology for non-profits with limited resources.

Nonprofit organizations like The Water Project face challenges in managing their tech infrastructure due to limited resources, requiring them to employ efficient solutions such as observability platforms to enhance their operational capacity. These platforms, particularly New Relic, enable small teams to monitor application performance and manage bugs effectively, ensuring minimal downtime and optimal performance for user interactions, which is crucial for donor engagement. The Water Project uses technology to gather project data, manage timelines, report spending, and collaborate with local communities to ensure access to water and sanitation. Observability tools help them address performance issues quickly and efficiently, allowing them to focus on their core mission while maintaining a lean engineering team. Additionally, through the Observability for Good Social Impact program, New Relic provides nonprofits with free and discounted access to its platform, demonstrating a commitment to equitable technology access.

Investing in observability tools has become essential for modern digital enterprises to deliver seamless digital experiences and drive innovation across various functions. However, identifying clear business outcomes and return on investment (ROI) has been a challenge. New Relic, as a leader in observability, understands that to realize its full potential, organizations must focus on maximizing uptime, optimizing business operations, and accelerating innovation. By doing so, businesses can improve system resilience, ensure peak performance, increase application security, optimize business operations, improve digital experiences, decrease cloud hosting costs, consolidate observability platforms, drive developer productivity, increase release velocity, and accelerate app modernization and AI adoption. With New Relic's value calculator, businesses can discover the potential ROI for their organization and see why the world's best teams rely on New Relic for business success.

In the evolving landscape of digital enterprises, observability tools are becoming crucial for enhancing digital experiences, optimizing operations, and fostering innovation, yet businesses often struggle to link these investments to clear outcomes and ROI. New Relic, a leader in this field, demonstrates how observability can maximize uptime by ensuring system resilience, optimizing business performance by reducing cloud costs and improving customer experiences, and accelerating innovation through faster developer productivity and AI integration. Case studies highlight significant achievements, such as William Hill's 80% improvement in incident recovery time and DAZN's ability to release 5,000 new codes daily, underscoring the impact of observability on business success. Arnie Lopez, New Relic’s Chief Customer Officer, emphasizes that observability not only maintains customer satisfaction and efficiency but also enables rapid adaptation to emerging technologies, thereby maintaining a competitive edge in the market.

At the heart of a successful digital enterprise is a commitment to customers, as a slow or unresponsive website can quickly lead to costly calls to customer service, users leaving without completing their purchases, and posting negative experiences on social media. Industry leaders recognize this, and investments in digital experience monitoring (DEM) technologies have surged over the past two years, with DEM deployments expected to rise from 60% to 90% by 2027 across all industry verticals. More than 16,000 global enterprises already leverage DEM solutions like New Relic DEM to enhance customer experiences and perfect their digital presence, achieving tangible return on investment (ROI). Leaders like Mercado Libre increased conversions by reducing website response time to less than 50 milliseconds, while The Seven Network achieved 100% uptime during live streaming of major events. DEM enables a smooth and efficient user experience, crucial for maintaining a competitive edge, by increasing customer satisfaction, boosting performance, delivering high-quality experiences, moving from reactive to proactive issue identification, gaining a competitive advantage, making better data-driven decisions, and reducing risk management. A fully-integrated, AI-powered DEM solution empowers businesses to monitor, analyze, and optimize digital experiences across all touchpoints, enabling organizations to adopt fast-evolving AI technologies like chatGPT with confidence. By leveraging integrated monitoring tools, businesses gain critical insights to eliminate interruptions in digital experiences before they happen, and can adopt AI with confidence, achieving digital success with New Relic DEM.

A commitment to customer satisfaction and maintaining a seamless digital experience is essential for the success of digital enterprises, as a slow or unresponsive website can lead to negative outcomes such as customer service calls, abandoned purchases, and damage to brand reputation. As a result, investments in digital experience monitoring (DEM) technologies have increased, with deployments expected to rise significantly by 2027. DEM solutions, such as New Relic, help businesses like Mercado Libre and The Seven Network improve performance by reducing website response times and ensuring 100% uptime during major events. Key benefits of DEM include increased customer satisfaction, optimized application performance, high-quality user experiences, proactive issue resolution, competitive advantage, data-driven decisions, and enhanced risk management. AI-powered DEM tools enable businesses to monitor digital experiences comprehensively, offering real user monitoring, session replay, synthetic monitoring, and crash analysis to identify and resolve issues quickly. These capabilities help organizations deliver consistent user experiences, adopt AI technologies confidently, and leverage insights for business growth.

Observability is crucial in today's complex IT environments to maintain reliable and efficient systems. There are three levels of observability maturity: basic, intermediate, and advanced. Basic observability focuses on essential metrics and simple alerting mechanisms, while enhanced observability incorporates more sophisticated metrics, logs, and traces. Advanced observability represents the highest level, where observability is deeply integrated into development and release lifecycles, including predictive analytics, automated root cause analysis, and proactive issue resolution. New Relic can be leveraged at each level to improve visibility, diagnose issues, and respond effectively to performance problems. By employing full-stack observability, AI-powered insights, and advanced alerting integrations, organizations can achieve enhanced reliability, improved performance, and proactive remediation, ensuring high system performance, better collaboration across teams, and an optimal digital experience for end users.

In the blog post by Zameer Fouzan, observability is highlighted as a critical component for maintaining robust IT systems by providing insights into their internal states through telemetry data analysis. The article outlines three levels of observability maturity—basic, intermediate, and advanced—each offering increasingly sophisticated techniques for system monitoring and performance improvement. At the basic level, monitoring focuses on essential metrics like CPU and memory usage, while intermediate observability incorporates more detailed metrics, logs, and distributed tracing to enhance application performance visibility. Advanced observability leverages predictive analytics and AI to anticipate and resolve issues proactively, ensuring minimal downtime and optimal user experience. New Relic's tools are presented as solutions to enhance observability at each level, providing features like infrastructure monitoring, application performance monitoring (APM), and AI-powered insights to support organizations in achieving higher system reliability and performance.

Alerts can be both beneficial and detrimental to software engineers, as they enable immediate intervention but also overwhelm with multiple alerts at once. Alert correlation is a strategic approach that groups related alerts together, enabling noise reduction, facilitated situational awareness, improved cross-party communications, and faster root cause identification. Correlation helps in identifying patterns of bursting alerts and how it can be applied to various scenarios such as alerts for the same entity, alerts between the same type of entities, alerts across various entities impacted directly or indirectly by the same source, and alerts between entities with relationships. Real-world scenarios demonstrate the effectiveness of correlation in reducing mean time to resolution (MTTR) and improving incident pattern matching procedures. To configure correlations effectively, filters can be applied to enforce isolation and custom tagging, generalization can be used to capture symptoms, and validation is crucial to ensure accuracy before applying rules in production.

Apache Airflow is an open-source scheduling tool and platform that enables users to create, schedule, and monitor workflows through Python code. It simplifies the process of managing complex workflows and automation pipelines across various industries, including ETL processes, machine learning pipelines, DevOps automation, and cloud infrastructure management. Workflows in Airflow are represented as DAGs (directed-acyclic graphs), which consist of tasks with dependencies that need to be completed in a specific order. Monitoring Airflow is crucial for ensuring its reliability and performance, and OpenTelemetry, an open-source observability framework, can be used to monitor and optimize Airflow workflows. By implementing OpenTelemetry, users can gain deeper insights into their workflows, diagnose issues faster, and ultimately optimize the performance and reliability of their data pipelines.

Apache Airflow is a crucial open-source tool for scheduling and orchestrating complex workflows in data engineering pipelines, but its effectiveness can be enhanced through proper monitoring, particularly with the use of OpenTelemetry (OTel), an open-source observability framework. Monitoring Airflow is essential for gaining insights, diagnosing issues, and optimizing pipeline performance; it involves tracking metrics like the health of components, DAG execution times, and pool utilization. OpenTelemetry offers a vendor-agnostic approach to monitoring, allowing users to export data to any backend and gain flexibility in data management. Implementing OpenTelemetry involves configuring the OpenTelemetry Collector and setting up Airflow to send metrics, which can then be visualized and analyzed in platforms like New Relic. This setup enhances the ability to troubleshoot and manage workflows proactively, ensuring improved reliability and performance of Airflow deployments.

Alert correlation is a strategic approach used by New Relic to manage and streamline the process of handling multiple alerts by grouping them into related clusters, thus improving the efficiency of identifying and resolving system issues. This method helps reduce noise by minimizing the number of alert notifications, enhances situational awareness by consolidating related information, and facilitates better communication through visual representations of incident timelines. By using correlation rules, such as grouping alerts by similar symptoms or common attributes, New Relic enables faster identification of root causes and shortens the mean time to resolution (MTTR). The article underscores the importance of configuring these correlations accurately to ensure effective incident management while avoiding unnecessary associations, thereby improving the overall observability and alert management process.