February 2019 Summaries

The "Relics of Color" Employee Resource Group at New Relic organized a Black History Month celebration, featuring panel discussions, happy hours, and film screenings to promote community, history, and insight. A panel discussion on Black Women in Tech brought together ten women from various teams across the company to share their experiences as part of an underrepresented group in tech. The panel highlighted similarities in their experiences, including feeling invisible or misunderstood, but also shared differences and ways they've overcome adversity. The event emphasized the importance of representation, citing how seeing people of color in leadership positions can inspire confidence and de-stigmatize being black in tech. The discussion encouraged women to use their voices to drive change and close the gap of representation in tech, with many panelists using their experiences to inspire others and build community. Ultimately, the event celebrated the accomplishments of Black Women in Tech and highlighted the importance of diversity, equity, and inclusion programs in creating a more inclusive industry.

In celebration of Black History Month, New Relic’s "Relics of Color" Employee Resource Group organized a series of events, including panel discussions featuring Black women in tech, to honor important events and figures in American history. The programming included appearances by notable speakers such as Emory Douglas, a Wakanda-themed happy hour, and a film screening of "BlaKkKlansman." One highlight was a panel where ten Black women from New Relic shared their experiences in the tech industry, discussing challenges related to underrepresentation and the importance of representation in leadership roles. The panelists emphasized the need for workplaces to align with values of diversity and inclusion, and they shared stories of empowerment and the impact of seeing people of color in leadership roles. Initiatives by panelists, like Mya James' non-profit in Ghana and Taylor West's mentorship with Black Girls Code, illustrate their commitment to closing the representation gap in tech. The events underscored the role of Black women as integral contributors to innovation in technology, advocating for continued change and recognition in the industry.

The New Relic Infrastructure agent can now be run as a non-root user, addressing security concerns for customers who cannot install and run it as root due to their security protocols. To enable this, the `nri-agent` user was created and Linux capabilities were defined to allow the user to collect metrics with elevated privileges. The agent offers two run modes: privileged and unprivileged, which provide different levels of access to system metrics and inventory. In privileged mode, the agent has total access to all system metrics and inventory, while in unprivileged mode, it can collect most metrics but lacks some inventory sources and process samples. Customers can install and configure the agent in these run modes using specific environment variables and installation instructions.

New Relic has introduced the ability for its Infrastructure agent to run as a non-root user, addressing customers' security concerns that restricted them from using the agent with root access. This led to the creation of the nri-agent user, which can operate in two modes: privileged and unprivileged. In privileged mode, the agent can access most metrics with elevated privileges, but does not enable Docker process metrics by default. In unprivileged mode, some metrics like file descriptor counts are not available, and certain inventory sources are not reported. Both modes require adjustments to custom integrations that originally depended on root access. The installation process involves setting the NRIA_MODE environment variable to choose between these modes, offering users flexible monitoring options that align with their security requirements.

In recent years, New Relic has faced challenges in ramping up release frequency without compromising quality. To solve this, they've adopted a microservices architecture and DevOps methodologies, which allowed them to increase deployment frequency from 20-70 times per day. However, this transformation required another critical capability: automating the code pipeline process. New Relic's approach emphasizes measuring and monitoring the pipeline to push changes to production with less risk. The company uses its platform to provide measurement, monitoring, and alerting capabilities that enable teams to identify and fix issues efficiently. By applying best practices such as capturing timestamped state changes, analyzing unit test results, and tracking successful deployments, teams can gain visibility and control over their code pipeline, accelerate development at scale, and deliver quality software.

New Relic has tackled the challenge of increasing release frequency without sacrificing quality by adopting a microservices architecture and DevOps methodologies, as explained in a webinar featuring Senior Solutions Manager Tori Wieldt. The transition from a monolithic structure to a more agile setup involved implementing over 200 microservices and deploying software up to 70 times a day, which was made feasible through advanced monitoring and instrumentation of their code pipeline. This approach allows for proactive management of deployments, ensuring both speed and quality in the development process. Essential to this transformation is New Relic's platform, which provides comprehensive monitoring and alerting capabilities across the entire development stack, enabling real-time insights and performance metrics. The webinar further explores best practices for optimizing code pipelines, including capturing data on source code changes, state transitions, and deployment success to enhance visibility and control over the development cycle.

The topic of this episode is microservices, addressing questions like what is a microservices architecture, best use cases for microservices, how to monitor them, trade-off between flexibility and complexity, and cultural implications of microservices. The guests are Richard Rodger, CEO of VoxGig, and Sean Carpenter, Principal Technical Evangelist at New Relic. They discuss the importance of understanding business logic, identifying sharable components, and isolating complexity in edge cases. Microservices primarily provide flexibility without complexity, but complexity is unavoidable. The key to success lies in having a clear definition of microservices, a strong separation of concerns, and isolating complexity into separate services that can handle the really ugly, messy stuff. Monitoring is critical for understanding system complexity, and distributed tracing plays a central role in this process. Microservices have a significant cultural impact, enabling remote working, diversity, and empowering individual teams to make decisions without needing permission from higher-ups. An acceptable error rate is also crucial for achieving velocity and making business decisions about risk tolerance.

The New Relic Modern Software Podcast episode delves into the intricacies of microservices, featuring insights from Richard Rodger, CEO of VoxGig and author of "The Tao of Microservices," and Sean Carpenter, a Principal Technical Evangelist for New Relic. The discussion covers the definition and benefits of microservices, emphasizing their role in enhancing flexibility and future-proofing business operations despite inherent complexity. The conversation also touches on the importance of monitoring and observability in microservices, with a focus on distributed tracing to manage system interactions and performance. Culturally, microservices support remote work and decision-making autonomy by enabling parallel workstreams and reducing the need for centralized approvals, ultimately fostering diversity and innovation within teams. The podcast underscores the trade-offs between development velocity and system complexity, advocating for strategic approaches to component composition and error management.

New Relic's FutureStack18 Global Tour was a series of events that brought together customers, partners, and experts in world-class cities. The tour has continued with the FutureStack19 RoadShows, which will visit Atlanta, Los Angeles, and Seattle. The YouTube playlist from FutureStack18 features 45 informative videos capturing keynotes and real-world customer stories, as well as hands-on technical presentations from New Relic experts and partners like Amazon Web Services. New Relic's views expressed on this blog are those of the author and do not necessarily reflect the views of New Relic. The solutions offered in the blog post are environment-specific and not part of commercial solutions or support offered by New Relic.

New Relic's FutureStack18 Global Tour was a series of engaging events held in major cities like San Francisco, London, and Sydney, bringing together customers, partners, and experts to share insights on modern cloud initiatives. The tour featured keynotes by New Relic founder Lew Cirne and RedMonk analyst James Governor, alongside real-world customer stories from companies such as IBM, Gap, Inc., and Atlassian. Attendees had the opportunity to gain hands-on experience through technical presentations by New Relic experts and partners like Amazon Web Services. For those unable to attend, a YouTube playlist with 45 informative videos is available, capturing the essence of the events. FutureStack19 RoadShows continue the tradition with upcoming stops in cities like Atlanta, Los Angeles, and Seattle, while interested individuals can stay updated via #FutureStack for new event announcements. The content reflects the author's views and not necessarily those of New Relic, with additional support available through the Explorers Hub for discussions related to the blog post.

Istio is an open-source service mesh that enables the connection, monitoring, and security of microservices deployed on-premise, in the cloud, or with orchestration platforms like Kubernetes and Mesos. It provides a layer of abstraction between application code and the underlying infrastructure, allowing for flexibility and scalability. Istio's architecture includes four main components: Envoy sidecar proxies, Mixer policy and telemetry hub, Pilot traffic management, and Citadel security layer. The service mesh provides features such as traffic management, circuit breaking, and load balancing, making it an attractive solution for modern software teams. While still in its infancy, Istio is gaining traction with a growing user base and support from major cloud providers.

Istio, an open-source service mesh developed by Google, IBM, and Lyft, addresses the complexity of managing microservices by providing capabilities such as service discovery, load balancing, and authentication. It acts as an infrastructure layer that enhances communication between service instances while offering a layer of abstraction that simplifies application management. Istio's architecture consists of a control plane and data plane, leveraging Envoy sidecar proxies and tools like Mixer, Pilot, and Citadel to manage traffic, security, and telemetry. Although still in its early stages of adoption, Istio has gained traction among tech leaders and companies like Ebay and AutoTrader UK, and is supported by platforms such as Kubernetes and cloud providers like Google Cloud. Its increasing popularity is driven by its ability to automate the management of microservices, providing flexibility and scalability necessary for modern applications. The integration with New Relic through the newrelic-istio-adapter further enhances its capability to collect and manage telemetry data, positioning Istio as a promising solution in the evolving landscape of microservices management.

Reliability is a complex issue in modern software development, with teams facing various challenges such as edge cases, variations across architectures and tiers, and the complexity of scale. To address these issues, engineering managers at New Relic use seven questions to determine if their teams are meeting reliability best practices, including robust deploy and rollback tooling, game-day testing, reliable pre-production testing, risk matrix updates, adequate free capacity, defensive rate limiting, and systems that can scale without meaningful architectural changes for the next 12 months. By regularly assessing these areas, teams can identify gaps and make improvements to enhance their reliability.

Reliability in software engineering is inherently complex, especially as teams contend with various challenges like defects, capacity issues, and operational debt, exacerbated by the scale of platforms like New Relic, which operates over 300 services and processes vast amounts of data. To maintain reliability, New Relic employs a set of best practices, guided by seven key questions that focus on areas such as ensuring bulletproof deploys and rollbacks, conducting game-days to test incident response, catching regressions before production, maintaining updated risk matrices, ensuring sufficient service capacity, implementing defensive rate limiting, and planning for scalability without major architectural changes. These practices emphasize the importance of continual improvement, as staying ahead of potential issues is crucial for maintaining system reliability and avoiding the pitfalls of falling behind in an evolving technological landscape.

To effectively monitor software at scale, organizations need to adopt automation best practices. At New Relic, they use four key strategies: automating monitoring agent installation in application builds, adding markers and tags to deploy systems, bootstrapping applications with monitoring, and using APIs and Domain Specific Languages (DSLs) for dashboards and alerting. By automating these tasks, organizations can reduce boilerplate code, eliminate manual configuration errors, and make it easier to manage complex microservices architectures. This approach not only improves monitoring efficiency but also reduces toil and uncertainty in operations, allowing teams to focus on innovation rather than manual monitoring tasks.

Operating applications at scale necessitates thoughtful and consistent monitoring strategies throughout the software lifecycle, from code deployment to alerting. New Relic adopts "monitoring as code" through automation, involving practices such as installing monitoring agents during application builds, using deploy markers and version tags for tracking, bootstrapping applications with templated builds, and utilizing APIs and Domain-Specific Languages (DSLs) for dashboarding and alerting. This approach reduces boilerplate code and enhances monitoring efficiency, enabling the company to scale its operations effectively while minimizing manual errors and toil. By integrating these practices, New Relic ensures comprehensive visibility and control over its applications and infrastructure as they continue to expand.

New Relic has acquired SignifAI, an event-intelligence company that focuses on correlating related issues and extracting predictive insights to enable software engineering teams to take action. The acquisition aligns with New Relic's movement towards a more open data platform and shares a vision for bringing machine intelligence to modern software operations. This aims to reduce the anxiety of being on-call for complex systems by providing context to human responders, reducing alert fatigue and allowing them to focus on higher-value work. SignifAI's technology sits above the monitoring tool layer, ingests alert data, and applies intelligence to it, telling users where to look to solve problems. The company's product and AI technology are informed by real-world site reliability engineering (SRE) problems, making it a great match for New Relic. The acquisition brings in a world-class team and one of the most powerful SaaS-based event correlation and intelligence platforms, enabling customers to turn more of their monitoring data into actionable insights.

New Relic has acquired SignifAI, an event-intelligence company specializing in correlating related issues and extracting predictive insights to help software engineering teams manage complex systems better. This acquisition aligns with New Relic's strategy to enhance its AIOps capabilities by leveraging SignifAI's technology, which integrates machine intelligence and site reliability engineering (SRE) best practices to provide actionable insights from diverse data sources. SignifAI's approach reduces alert fatigue and helps teams focus on creative problem-solving by offering a "Virtual SRE" that efficiently identifies root causes of issues amidst the noise of multiple alerts. This acquisition aims to support New Relic's vision of an open data platform where users can benefit from intelligent monitoring solutions regardless of the data source, fostering faster and more informed decision-making. The collaboration between New Relic and SignifAI is expected to advance automated diagnostics and self-healing systems, addressing the growing complexity in modern software operations.

This guide presents best practices and insider tips for getting the most value from New Relic Alerts, a key feature of the New Relic platform that helps teams stay informed about potential performance issues before they occur. By setting up policies and conditions, choosing and using incident preference, setting up notification channels, understanding what is displayed in New Relic Alerts, taking action on alerts, sending alerts data to Insights, and configuring baseline alerts, teams can minimize the risk of "alert fatigue" and ensure that they receive notifications that are relevant and timely. The guide also covers topics such as learning the language of alerting, exploring incidents, and using webhooks to create custom notifications. By following these best practices, teams can get the most out of New Relic Alerts and improve their incident response process.

New Relic Alerts is a crucial tool for monitoring potential performance issues in real-time without the need for constant manual oversight. It automatically notifies team members when pre-defined alert conditions are triggered, minimizing the risk of alert fatigue that can lead to errors in incident response. The platform offers features like incident rollups and prioritized search terms to streamline notifications and enhance team communication. Users can configure alert policies, conditions, thresholds, and notification channels tailored to specific needs, ensuring that the right team members are informed promptly. Incident preferences allow for customizable notification frequencies, addressing the challenge of redundant alerts and enabling teams to maintain focus on critical issues. The incident context feature provides a detailed analysis of performance anomalies at the time of alert, facilitating quicker diagnosis and resolution. Moreover, alerts data can be integrated with New Relic Insights for advanced analysis, and baseline alerts can be configured to adapt to performance variations over time. This comprehensive setup helps teams optimize their alert systems and improve overall incident management efficiency.

The SMART by GEP engineering team built a custom measurement solution using New Relic to monitor their code development lifecycle, following the Best Practices for Measuring Your Code Pipeline example. The team measured six key metrics: commits based on product and branch, commits per user, files modified per commit, comparison of commits, and lines of code changed. They used New Relic's code pipeline example as a starting point and adapted it to Microsoft Azure instead of Amazon Web Services (AWS). The team created an Azure Function that consumes a GitHub webhook, formats and sends data to the Insights API, and runs NRQL queries to visualize the data in a dashboard. By tracking these metrics, the team gained insights into developer time usage, efficiency, code quality, and branching issues, which helped them reduce inefficient code practices and improve their development process. The project demonstrates the versatility of the New Relic platform and its ability to be used for innovative applications beyond standard use cases.

An engineering team at GEP utilized New Relic's platform to enhance their code development lifecycle after attending a presentation on measuring code pipelines at a FutureStack event. The team developed a custom measurement solution to monitor six key metrics, such as commits per branch and files modified per commit, using New Relic's insights and AWS services like CodePipeline and Lambda functions. Although GEP's team operates on Microsoft Azure, they successfully adapted the pipeline to Azure with New Relic's assistance. Their GitHub Commits dashboard provides a comprehensive view of code changes, efficiency, and code health, enabling the team to track developer activities and streamline processes by reducing unused branches. This initiative highlights New Relic's flexibility in extending its monitoring capabilities beyond standard practices, allowing developers to perform advanced analytics and improve code-pipeline practices across GEP.