August 2022 Summaries

Tail sampling is a technique used to identify issues in distributed systems while reducing observability costs. It involves controlling which spans are sent to an observability backend, resulting in lower ingest costs. Tail-based sampling, also known as headless sampling, occurs after all spans in a request have been completed, allowing for more flexibility in filtering traces based on specific criteria. To implement tail sampling using the OpenTelemetry collector, a component called the tail sampling processor must be used, which samples traces based on policies that can be chosen from and defined. The configuration of this processor includes settings such as decision wait time, number of traces to keep in memory, and expected new traces per second, as well as policies for filtering spans by status code or probabilistic sampling. While tail sampling offers benefits, it also has limitations, including unpredictable costs, performance issues, and challenges in figuring out the right policies. Additionally, OpenTelemetry has some limitations, such as requiring a collector deployment pattern and load balancing, and an evolving project with ongoing development of its components. Alternatively, New Relic's Infinite Tracing option can be used for tail sampling, offering a simpler configuration process and integration with OpenTelemetry.

Tail sampling is a method used in distributed systems to manage observability costs and focus on important data by sampling traces after all spans in a request are completed, contrasting with head-based sampling, which occurs at the beginning of a request. This approach allows for filtering based on specific criteria, such as error traces or latency, reducing the amount of data stored and processed. Implementing tail sampling in OpenTelemetry involves using the tail sampling processor, which relies on policies to determine which traces to sample, though it presents challenges such as unpredictable costs and the need for a suitable collector setup. New Relic's Infinite Tracing offers an alternative solution by simplifying the process through a trace observer that supports OpenTelemetry and New Relic's APM agents, enabling users to configure sampling rules without altering their existing infrastructure. Despite its advantages, tail sampling requires careful consideration of its limitations and potential issues, including resource allocation and the possibility of fragmented traces due to incomplete data collection.

The new Data Plus option from New Relic provides engineering teams with improved observability capabilities at a simple and affordable per-GB pricing model, addressing maturing needs with extended data indexing retention, logs obfuscation, FedRAMP, and more. This option includes longer query durations for NRQL queries through NerdGraph, allowing customers to run queries up to 10 minutes without timing out, making it suitable for complex data volumes and reducing mean time to detection (MTTD) and lower visibility due to failed queries. By utilizing async: true in NerdGraph queries, users can reduce query interruption risk and retrieve results at a later time, and the option also provides bundled savings of up to 56% compared to the original data option. To take advantage of these extended query limits, customers can start with the NRQL query builder or dive into the NerdGraph API, and New Relic offers a free trial account for those who are new to the platform.

Data Plus is a new option that enhances observability for engineering teams by offering affordable per-GB pricing and improved capabilities such as extended data indexing retention and logs obfuscation. This option includes longer query durations, allowing NRQL queries to run for up to 10 minutes through NerdGraph and standard queries for up to 2 minutes, facilitating the search through large datasets without timing out. It is particularly beneficial for complex business systems, enabling users to build custom dashboards and alerts. By using the NerdGraph API with Data Plus, users can execute long-running queries asynchronously, which reduces the risk of interruptions and allows results retrieval at a later time. This offering is priced at $0.50 per gigabyte, with potential savings of up to 56%, and encourages users to upgrade for better performance in larger data environments.

Java performance monitoring is essential for ensuring that Java applications run smoothly, efficiently, and meet performance requirements. Poorly performing Java applications can have a significant negative impact on various areas of business, from customer satisfaction to employee productivity. There are several types of Java monitoring tools available, including basic JVM monitoring tools, profiling tools, application performance monitoring (APM) solutions, metrics and analytics platforms, logging and log analysis tools, memory analysis tools, thread analysis tools, and real-time monitoring and alerting tools. To improve Java application performance, developers can optimize their Java code by avoiding recursion, leveraging StringBuilder, using stack allocations instead of heap allocations, choosing the right garbage collector, and identifying and fixing slow critical regions. Additionally, a good monitoring tool provides actionable real-time insight into how an application is performing, including observability into JVMs running the application, detailed insights into slow transactions, and proactive alerting if issues arise.

Logging is crucial for understanding applications and services, but effective log management can be overwhelming due to the volume of data and configuration complexity. To overcome this, it's essential to carefully select what to log, plan for common logging scenarios, and use meaningful messages that drive decisions. Log rotation and retention policies are also critical for maintaining system performance, regulatory compliance, and effective data management. Additionally, using standardized log formats, consistent log levels, and timestamps can simplify analysis and troubleshooting. By implementing these best practices, developers can gain visibility into their stack, troubleshoot errors faster, and better understand user experience.

Effective log management is essential for understanding the applications and services you develop, offering insights into system performance and error diagnosis. Key practices include carefully selecting what to log, ensuring logs contain valuable metadata without excessive data, and using clear, consistent formats to facilitate parsing and analysis. Log rotation and retention policies help manage storage efficiently while adhering to compliance standards. Including timestamps and applying log levels such as DEBUG, INFO, WARNING, ERROR, and CRITICAL categorizes the severity of messages for efficient troubleshooting. Tools like New Relic can enhance log management by allowing customizable parsing rules and offering features to integrate logs with telemetry data, ultimately improving system maintenance and decision-making processes.

Operational efficiency is a key concept in achieving business goals through effective resource allocation and utilization. According to the 2021 Accelerate State of DevOps report by Google Cloud's DevOps Research and Assessment (DORA), teams that excel at modern operational practices are more likely to report better software delivery, operational performance, and business outcomes. To improve efficiency as part of an observability practice, it is essential to collect all telemetry data in one place, establish a common language among team members, automate instrumentation and observability, and reign in tool sprawl by consolidating relevant tools and systems into a single source of truth. By implementing these strategies, teams can quickly resolve issues, proactively identify performance optimization opportunities, and achieve operational efficiency.

Hex Technologies is a platform for collaborative data science and analytics that brings together SQL, Python, R, and no-code workflows in powerful notebooks to facilitate exploratory analysis with minimal friction and publish projects as interactive data apps that anyone in an organization can use. The company aims to bridge the gap between data scientists and non-technical users by offering a "low-floor, high ceiling" approach that pairs code with powerful UI tools. Hex has gained significant traction and raised its Series B financing led by Andreessen Horowitz, Databricks, Snowflake, and existing investors. Despite the challenges of building a company from scratch, the founders have emphasized the importance of collaboration, documentation, and tooling to ensure the success of their platform. They attribute their reputation for shipping quickly and their ability to scale an effective engineering team to New Relic's fast, intuitive, and customizable UI and flexible pricing. The company has seen significant benefits in using New Relic, including improved debugging capabilities, faster log searching, and easier collaboration among teams.

Operational efficiency is crucial for DevOps, operations, and site reliability engineering teams to achieve effective outcomes by optimizing the ratio between business inputs and outputs. The 2021 Accelerate State of DevOps report highlights that teams excelling in modern operational practices show significantly better software delivery and business results. To enhance efficiency in observability practices, four strategies are recommended: consolidating telemetry data in one platform, establishing common naming conventions for effective communication, automating observability processes, and reducing tool sprawl for streamlined operations. These approaches help in quickly resolving issues and optimizing performance while lowering operational complexity and costs. The insights are shared by Neil MacGowan, Director of Solutions Architects at New Relic, who emphasizes that the solutions discussed are specific to certain environments and are not officially part of New Relic's commercial offerings.

Hex Technologies, co-founded by Glen Takahashi, Barry McCardel, and Caitlin Colgrove, offers a collaborative data science and analytics platform that integrates SQL, Python, R, and no-code workflows into user-friendly notebooks, enabling both technical and non-technical users to perform exploratory analysis and create interactive data applications. The founders, motivated by their own experiences with data analytics, aim to bridge the gap between data scientists and non-technical users. Hex's solution features real-time collaboration, no-code charts, first-class SQL support, and a reactive, graph-based compute engine, emphasizing a "low-floor, high-ceiling" approach to make data analytics accessible and impactful. The company has faced challenges such as building comprehensive documentation and tooling to support a full-stack engineering approach, but it has maintained a reputation for rapid product development, thanks in part to utilizing New Relic's fast and intuitive UI for monitoring and debugging. The platform raised Series B financing with support from notable investors like Andreessen Horowitz, Databricks, and Snowflake, underscoring its potential in the evolving modern data stack landscape.

The company, New Relic, is undergoing a restructuring process to ensure long-term growth and delivery for its customers. The first quarter of 2023 showed strong results but identified the need to be more efficient with resources. Non-essential program expenditures, tools, external contracts, and travel were reduced, and an organizational assessment was conducted to identify role redundancies and misalignment with strategic priorities. As a result, approximately 90 US-based employees will be let go, representing less than 5% of the company, while up to 20 international employees may also be impacted. The company is providing support and severance packages to departing employees, including extended benefits coverage, career outplacement support, and laptop retention. The goal is to maintain focus on building a great business while supporting affected team members through this transition.

OpenTelemetry aims to standardize telemetry data collection and sending to a backend platform, providing a world-class observability experience in New Relic using any telemetry source, including hundreds of supported data sources. However, some experiences may differ due to the newer technology and different data model between OpenTelemetry and New Relic agents. A detailed feature comparison is available on the New Relic documentation site, which will be continuously updated with more support for both OpenTelemetry instrumentation and specifications. The author encourages feedback on desired features and invites users to explore the Explorers Hub forum for questions and support related to OpenTelemetry and New Relic.

Interest in OpenTelemetry is increasing, and a new feature comparison between New Relic agents and OpenTelemetry data has been published to address how observability experiences differ when using these technologies. OpenTelemetry, supported by the Cloud Native Computing Foundation, aims to standardize telemetry data collection, and New Relic's goal is to deliver a consistent observability experience regardless of the telemetry source. Although OpenTelemetry is advancing quickly, there are still aspects of the New Relic platform that it doesn't support yet. The comparison, which will be continually updated, is available on New Relic's documentation site, and users are encouraged to provide feedback and suggest changes. New Relic is committed to expanding support for OpenTelemetry and New Relic platform features, and users can submit feature requests through the Explorers Hub forum.

New Relic's CEO, Bill Staples, announced a targeted restructuring to ensure the company remains aligned with growth trends and market expectations, resulting in a workforce reduction of about 5%, affecting approximately 90 employees in the U.S. and up to 20 internationally. The decision followed a reassessment of the FY23 budgets and a thorough organizational evaluation to identify non-essential expenditures and role redundancies. The company is committed to supporting affected employees with severance packages, extended benefits, career outplacement support, and other transition aids. Staples expressed gratitude for the contributions of departing employees and emphasized the company's ongoing mission to empower engineers and maintain commitment to its core values.

The Java programming language remains incredibly popular, with Python trailing it in popularity, according to a recent index. The latest release of Java 17 has shown significant growth since its initial release, surpassing non-LTS versions like Java 6, Java 10, and Java 16. The open source evolution of Java has led to a shift away from Oracle's JDKs, with Amazon Corretto growing in popularity, alongside other vendors such as Eclipse Adoptium and Azul Systems. Containers have become an essential element in cloud computing, with over 70% of Java applications reporting to New Relic running in containers, leading to changes in how teams use their computing resources and memory settings.

Java continues to be a dominant force in the software development world, remaining the second-most popular programming language globally according to the Popularity of Programming Language Index. The recent release of Java 17, the first long-term support (LTS) version since Java 11, marks a significant milestone, although its adoption is still in the early stages compared to Java 11, which has seen widespread use since its release. The evolution of Java as an open-source platform has contributed to a democratized market, with Oracle's dominance decreasing as competitors like Amazon Corretto, Eclipse Adoptium, and others gain traction. The proliferation of containerized applications, which over 70% of Java applications now leverage, reflects a broader trend in cloud computing that emphasizes efficient resource utilization. These developments underscore Java's integral role in shaping modern software development practices, as highlighted in New Relic's 2022 State of Java in the Ecosystem report.

Cross-browser testing is crucial for ensuring that web applications and websites work properly across different browsers, devices, and assistive technologies. It involves determining if the website or application can be accessed through various combinations of browsers, operating systems, and devices. With the constant changes in browser functionalities and updates, cross-browser issues can arise, leading to slow page times, errors, and poor user experience. To address this, developers need to adopt a holistic approach to testing, including manual and automated methods, to ensure that their application works as expected on all browsers and devices. By implementing cross-browser testing, developers can gain peace of mind knowing that their web application or website is compatible across multiple browsers, old and new. Tools like New Relic can provide real-time insights into user experience, helping developers identify and address errors quickly.

Veygo, a car insurance company, adopted serverless architecture to optimize IT resources and understand customer behavior in real-time, particularly during peak traffic times and holidays. They integrated Salesforce customer data across mobile and web applications using AWS Lambda functions, allowing customer agents to access this information instantly without soliciting it from customers. To monitor their serverless infrastructure, Veygo built dashboards that focus on core metrics, such as throughput, deployment frequency, lead time for change, stability, and mean time to recovery (MTTR). These dashboards help the company track its performance and identify areas for improvement, with a goal of achieving 200 releases per month and an MTTR within an hour.

Veygo, a company in the short-term car insurance industry, utilizes serverless architecture to efficiently manage IT resources and customer data, particularly during peak traffic times and holidays, without the cost of maintaining excess servers. This approach allows Veygo to integrate Salesforce customer data from mobile and web applications seamlessly across operations, enhancing real-time customer interactions through automated AWS Lambda events. The use of serverless infrastructure, supported by the Serverless Framework and AWS services like API Gateway, WAF, and X-Ray, facilitates an agile development process with a focus on continuous deployment and monitoring through dashboards. These dashboards, integrated with New Relic, provide insights into API usage, Lambda functions, and error rates, enabling Veygo's software engineers to maintain a lean, agile culture and improve deployment efficiency. The company tracks key metrics like deployment frequency, lead time for change, change failure rate, and mean time to recovery (MTTR) to balance efficiency and quality in their CI/CD pipeline, with a commitment to maintaining high standards of infrastructure resilience and customer experience.

PHP is a widely-used programming language for server-side applications, but it's also vulnerable to various security threats, including cross-site scripting (XSS), SQL injection, cross-site request forgery (CSRF), session hijacking, buffer overflows, denial of service (DoS) and distributed denial of service (DDoS) attacks, password cracking, and man-in-the-middle (MitM) attacks. To improve PHP security, it's essential to incorporate vulnerability management tools like New Relic, follow secure coding practices, assume that attacks will happen and be prepared for them, use secure hosting, make systems fault-tolerant, conduct security audits, monitor activity in the application, set up alerts for critical performance thresholds, protect databases, and parameterize queries. By taking these steps, developers can reduce the risk of security breaches and protect their PHP applications from various types of attacks.

In this article, monitoring three network devices was discussed, including setting up a simple network configuration and using New Relic to monitor them. The author assumes that readers are familiar with Linux and Docker setup, and provides detailed instructions on how to set up the network devices and configure EIGRP routing protocol. The author also guides readers through the process of installing the New Relic virtual machine, setting up the agent, and configuring SNMP for monitoring. Once the setup is complete, readers can monitor their network devices using New Relic's dashboard, which provides high-level information about CPU, RAM, and individual interface stats, as well as other screens and dashboards depending on the device configuration.

Leon Adato's blog post guides readers through the process of setting up and monitoring a network configuration involving three routers using New Relic's tools. The post details the configuration of routers named Spring, Summer, and Fall, explaining how they are connected in a triangular layout using the EIGRP routing protocol and a common switch to facilitate communication. The routers are configured via commands entered through a terminal emulator like PuTTY, and the post emphasizes noting differences in IP addresses and hostnames. The article further explains how to integrate these network devices with New Relic for monitoring purposes, including setting up the necessary software on a virtual machine, configuring the SNMP protocol, and validating the setup to ensure proper monitoring of network performance. Adato encourages readers to leverage New Relic's capabilities for enhanced network observability, offering practical advice for both beginners and experienced users, and invites them to join the New Relic community for additional support and resources.

Monitoring is a crucial aspect of managing complex systems, and alerts play a vital role in this process. An alert is a notice to take immediate action regarding an issue, but it should be designed to respect employees' time, provide context, and be urgent and non-optional. Alerts that don't require human intervention are not alerts, but rather automation or reports. A good monitoring tool should enable the creation of event flags that feed back into the monitoring system, providing valuable context for resolving issues. To improve alerts, it's essential to have an open conversation with the recipients, reviewing and adjusting trigger conditions and actions as needed. Effective alerting is not just about generating alerts, but also about understanding how they relate to observability and monitoring.

The blog post by Leon Adato explores the complexities of monitoring and alerting systems, emphasizing the importance of creating alerts that are actionable, context-rich, and respectful of human resources. Drawing from personal experience, Adato highlights the potential pitfalls of poorly configured alerts, such as generating excessive tickets without providing meaningful insights or solutions. He argues that effective alerts should not only notify but also facilitate problem-solving by offering relevant context and should only trigger human intervention when necessary. Furthermore, Adato underscores the distinction between monitoring, which involves the systematic collection of data, and observability, which relates to the analysis and understanding of that data. He stresses the need for continuous evaluation and adaptation of alert systems to ensure they remain effective as applications evolve, urging practitioners to engage in ongoing dialogue with those affected by alerts to prevent them from becoming mere background noise.