March 2019 Summaries

You might not think of the United Kingdom as a red-hot market opportunity for pizza delivery. But on the Friday before Christmas of 2018, Dominos Pizza UK sold a record 535,000 pizzas, according to The Guardian—cranking out an average of 12 pizzas a second for millions of hungry Britons. Dominos Pizza UK relies on its e-commerce platform to deliver rock-solid performance at massive scale while ensuring customers get highly responsive digital experiences. New Relic Insights are critical in maintaining this focus by providing visualization and helping the company grasp trends impacting conversions and customer satisfaction. The platform helps Dominos instantly understand how customers want to engage with them as a digital brand, enabling the delivery of satisfying experiences and delicious pizza to its customers.

In December 2018, Domino's Pizza UK achieved a record-breaking sale of 535,000 pizzas in one day, necessitating the robust performance of its e-commerce platform to manage high-volume online orders and deliveries. To ensure exceptional digital customer experiences, Domino's relies on the New Relic platform, with a particular emphasis on New Relic Insights for real-time data visualization, helping the company understand and respond to customer engagement trends. Amjid Chohan, the software development manager for Domino's Pizza Group, highlights the importance of these insights in maintaining customer satisfaction and meeting evolving digital expectations. The platform's visualization capabilities are crucial for identifying patterns that affect conversions and satisfaction, demonstrating how Domino's leverages technology to enhance both service and customer experience.

New Relic Synthetics offers proactive website and app monitoring in real-time from 18 locations worldwide to help companies catch problems before customers notice them. The platform has recently added features that make it easier to track and manage changes in synthetic scripts, as well as provide insights into who is using the platform and how. New Relic audit logs can now capture critical data about these changes, including user activity, script updates, and monitor downtime. These audit logs help companies keep up with the pace of change in their modern enterprise by providing a human-readable description of the action taken and a timestamp for the change. The secure credentials API allows teams to automate Synthetics access and modify sensitive information globally via the API, saving time and effort when updating passwords or other critical user information.

New Relic Synthetics has introduced new features aimed at improving the management and monitoring of website and app performance through audit logs and a secure credentials API. These enhancements allow organizations to track changes made by users to synthetic monitoring scripts, ensuring transparency and accountability by providing detailed logs that include who made changes and when. This functionality is particularly useful for version control, enabling rollbacks if necessary, and helps prevent future issues by identifying the source of problematic changes. The secure credentials API automates the process of updating sensitive information like passwords across multiple scripts, reducing manual effort and increasing efficiency by allowing updates to be made globally with a single command. These tools support the seamless operation of synthetic monitoring, which is crucial for maintaining the uptime and performance of web services in dynamic enterprise environments.

In a modern technology stack, complexity can lead to fragility, making incident response processes crucial. However, teams often struggle with collaboration and retrospectives, which are essential for learning from incidents. New Relic's partnership with the SNAFUcatchers project aims to improve incident response by recognizing the importance of teamwork, process-tracing, and maximizing the value of retrospectives. By avoiding the post-mortem "cycle of death" and focusing on learning from incidents, teams can uncover the most pressing sources of risk within their systems and drive meaningful improvements.

A recent New Relic FutureTalk event highlighted the challenges and strategies in managing incident response within complex technology systems, emphasizing the importance of learning from incidents to enhance system reliability. The discussion featured insights from Beth Long, a DevOps Solutions Strategist, and Tim Tischler, a Site Reliability Champion, who stressed that incident response is a collaborative effort often involving cross-team coordination, akin to the roles of first responders and medical professionals. They warned against the pitfalls of the "post-mortem cycle of death," where retrospectives become routine rather than meaningful learning experiences, and advocated for deep dives into incidents that involve multiple participants and interesting scenarios. The session also highlighted the dangers of oversimplifying incident causes, such as attributing failures to human error, without considering systemic issues. Long and Tischler recommended conducting timely retrospectives to capture accurate details and urged organizations to understand that while complex systems are inherently unknowable, incidents can reveal key areas for risk reduction and improvement.

At New Relic, the company employs reliability best practices through chaos engineering to test its systems for efficiency and resiliency against hazardous conditions, using techniques like carefully injecting harm into systems to prepare for outages. The company leveraged its GraphQL API as an entry point for internal chaos engineering practices and developed a tool called Chaos Panda to test against latency and field errors in the API. With Chaos Panda, teams can configure the API to add latency or cause certain fields to fail at specific failure rates, allowing them to identify opportunities to improve resilient code in their services. The tool is designed to be simple and easy to use, with a mutation that kicks off a chaos session for specific fields and slows down responses by a specified amount of time.

New Relic is enhancing the resilience and efficiency of its systems by employing chaos engineering and adversarial gamedays, focusing on controlled and thoughtful experiments to understand system behavior during failures. The Unified API team has integrated chaos engineering practices into their GraphQL API, utilizing a tool called Chaos Panda to intentionally disrupt system responses, such as adding latency or causing field errors, to test and improve system robustness. GraphQL is favored for its ability to manage complex queries through a single endpoint, unlike traditional REST APIs. Chaos Panda allows for pre-production testing by configuring GraphQL to simulate errors and delays, providing insights into partial response handling and opportunities to strengthen the code against potential failures. This approach is part of New Relic's broader initiative to simplify and enhance the development and testing processes for their developers, marking the beginning of their journey in using the GraphQL API for chaos engineering.

At New Relic, a company that helps customers deliver great customer experiences by monitoring their systems in real-time, exceptional people build exceptional products, and the scale of the company is unparalleled, with over two billion events and metrics per minute. The company's mission is to provide value to its customers, who include more than half of the Fortune 100, and to hire smart and interested individuals who can solve hard problems and learn from each other in a dynamic environment. The company is just getting started and is looking for talented engineers to join its development centers around the world.

New Relic Chief Product Officer Jim Gochee highlights the unique appeal of an engineering career at New Relic, emphasizing the unparalleled scale of operations, processing over two billion events and metrics per minute, which few companies experience. This scale is crucial for delivering exceptional customer experiences, especially for over half of the Fortune 100 companies that depend on New Relic during critical moments. Gochee stresses the importance of hiring smart individuals who are eager to tackle challenging problems, ensuring they are placed in environments conducive to growth and learning. With continuous expansion and new projects, New Relic offers numerous opportunities for involvement and emphasizes its ambitious mission as a company. The blog post also invites potential candidates to explore career opportunities in various global locations and provides a video for further insight, with a disclaimer that the views expressed are those of the author and do not necessarily reflect those of New Relic.

The distributed tracing team at New Relic used the Andon system to resolve a bug that impacted the accuracy of traces, which they called "the span count bug." The team first activated an Andon status to signal for help and then worked together to identify the root cause of the issue. They found that a service called "The Trace Indexer" was using a RocksDB instance with a cache layer on top, which was causing the problem. To fix it, they resized the cache to prevent it from evicting records before aggregations were complete and pursued a Kafka Streams upgrade as a permanent fix. The team learned several key takeaways from their experience, including declaring priorities, engaging outside support early and often, increasing communication inside and outside the team, prioritizing troubleshooting steps, accepting a higher risk tolerance, being willing to look at big changes, using data, documenting evidence, and avoiding duplicating efforts. Overall, the Andon system helped the team resolve the issue efficiently and effectively, while also improving their communication and collaboration processes.

New Relic's incident response process includes the use of the Andon system, originally developed by Toyota, to manage non-emergency issues that might impact customer experience, such as the "span count bug" in their distributed tracing team. The Andon system allows teams to signal their status—green for OK, yellow for warning, and red for emergency—through a dedicated Slack channel, enabling increased support from other teams and resources. When the distributed tracing team discovered a discrepancy in trace data, they went "Andon red" to prioritize resolving the issue, involving extensive troubleshooting steps, engaging external support, and conducting a blameless retrospective to capture nine key takeaways from the process. These takeaways include prioritizing central issues, increasing communication, accepting manageable risk, and documenting evidence, highlighting the importance of a structured Andon process in efficiently addressing and resolving system problems to prevent future customer dissatisfaction.

The Complexity in Context video series by New Relic aims to provide a clear and hands-on understanding of how to use New Relic to cut through complexity in modern software environments. The first episode focuses on observability, particularly in microservices-based environments, where complex relationships between services can make troubleshooting challenging. To address this, New Relic offers features such as distributed tracing, benchmarking, and a service map that help developers understand the relationships between services and identify performance issues. By using these tools, developers can gain a better understanding of their systems and teams, leading to smoother collaboration and faster issue resolution.

The "Complexity in Context" video series by New Relic is designed to help users navigate the complexities of modern software environments, particularly as they transition from monolithic to microservices-based architectures. The inaugural episode focuses on the evolution of observability and provides a practical example of using New Relic to troubleshoot performance issues in microservices environments. The process involves four main steps: obtaining an overall view of application performance, using distributed tracing to identify performance issues, benchmarking to pinpoint anomalous spans, and gathering detailed data to resolve these problems. The example highlights the interconnected nature of services within a distributed system and underscores the importance of understanding these relationships to effectively diagnose and fix performance bottlenecks, such as a database issue discovered through New Relic's tools. The content, written by Matthew McKenzie, emphasizes how New Relic's observability capabilities can enhance the efficiency of teams in managing complex software environments.

New Relic`, a technology company, has partnered with `Innovate Oregon` and the `Dayton Public Schools` to test the application of Agile principles in the classroom. The project aimed to empower teachers with new tactics to engage students, make learning more relevant and rewarding, and help teach students how to think for themselves. With the support of New Relic's Agile practitioners, the partnership successfully launched an initial effort to bring Agile thinking to the classroom in Dayton, Oregon, and later scaled it across multiple classrooms. The project has shown promising results, with teachers reporting increased student engagement and improved problem-solving skills. The initiative seeks to replicate its success and bring Agile methodologies to school systems across Oregon and the nation, providing a unique opportunity for technology professionals to make a meaningful impact on education.

Agile thinking, widely successful in the technology industry, is being explored as a transformative approach in education through a collaboration between New Relic, Innovate Oregon, and Dayton Public Schools in Oregon. Spearheaded by Thompson Morrison and supported by New Relic volunteers, the initiative aims to apply Agile principles—such as feedback loops, iterative cycles, and the empowerment of teachers—to create more engaging and relevant educational experiences. The program, initiated in Dayton, Oregon, has seen promising results, with teachers reporting enhanced classroom dynamics and improved problem-solving skills among students. This collaboration not only offers a new perspective on managing classroom challenges but also serves as a rewarding experience for the participating technology professionals, who find value in using their expertise to impact education. The success of this pilot program has sparked interest in scaling the model to other schools, with calls for more Agile practitioners to join the effort, demonstrating a potential path for integrating modern business practices into educational systems nationwide.

The DORA team has been conducting an annual survey and report on the state of DevOps for five years, gathering insights from over 23,000 professionals to track industry trends and share best practices. The 2019 report aims to build on this research by including contributions from participants in a new survey. A key finding from previous reports is that elite DevOps performers have a significant advantage in terms of lead times, change failure rates, and time to recover from incidents. To contribute to the 2019 report, professionals are encouraged to take a 25-minute survey, which will help shape the industry's understanding of DevOps best practices. The DORA team, including Google and New Relic, sponsors the research to advance the field of DevOps and software team performance.

DORA's Accelerate: State of DevOps report is a crucial resource for understanding trends in the DevOps industry, providing insights into best practices such as monitoring, observability, continuous testing, database change management, and early security integration. Sponsored by entities like Google, the report distinguishes elite DevOps performers from less successful ones, highlighting their ability to deploy software multiple times a day with swift lead times and rapid recovery from outages. The 2018 report noted that elite performers achieved significantly more frequent code deployments, faster lead times from commit to deploy, lower change failure rates, and quicker incident recovery times. DORA encourages participation in its annual survey, which has historically gathered input from over 23,000 professionals, to continue refining the industry's practices. The insights from these reports are encapsulated in the book "Accelerate," which serves as a guide for DevOps transformations.

This post highlights nine common coding problems that are often better solved using existing solutions rather than trying to code them from scratch. These problems include parsing HTML and XML, CSV files, JSON data, email address validation, URL processing, date/time manipulation, templating systems, logging frameworks, and security/encryption. The author emphasizes the importance of reusing existing libraries and tools to save time and reduce errors, rather than trying to create custom solutions from scratch.

Originally published in 2014 and updated in 2019, this blog post by Andy Lester discusses the importance of utilizing existing libraries and solutions for common coding challenges to save time and reduce errors. It highlights nine specific tasks, such as parsing HTML, CSV, and JSON data, email validation, URL processing, date/time manipulation, templating, logging, and encryption, where using pre-built libraries is preferable to coding from scratch. The post emphasizes the complexity and potential pitfalls involved in these tasks, arguing that leveraging well-established libraries can prevent bugs and inefficiencies. Lester also underscores the significance of allocating a programmer's time wisely, noting that the integration of additional libraries has minimal impact on execution time compared to the time spent on debugging custom code. The article concludes by reminding readers that while creating code can be satisfying, often the most efficient solution is to rely on pre-existing tools and libraries developed and tested by experts.

The Lesbians Who Tech (LWT) organization is a seven-year-old initiative dedicated to increasing visibility and representation of queer women in tech. The LWT Summit, held annually in San Francisco's Castro neighborhood, boasts excellent speakers on diverse topics, while also offering a friendly and inclusive community atmosphere. With over 40,000 members worldwide, the organization provides various resources, including the Edie Windsor Coding Scholarship and include.io, which connects underrepresented minorities with mentors and hiring companies. New Relic was proud to sponsor the event, allowing them to connect with attendees about their company's commitment to diversity and inclusion in tech. The summit featured inspiring speakers, such as Kara Swisher, Laurene Powell Jobs, Senator Tammy Baldwin, London Breed, Alicia Garza, Stacey Abrams, Danielle Feinberg, and Rachel Etnire, among others. The event is set in the historic Castro Theater, a venue that holds significance for its connection to Harvey Milk, a pioneering gay activist. Overall, the LWT Summit is an essential conference for anyone interested in tech, diversity, equity, and inclusion.

Lesbians Who Tech (LWT) is an organization focused on enhancing visibility and representation for queer women in the tech industry, hosting one of the largest LGBTQ professional events globally, the LWT Summit in San Francisco's Castro neighborhood. The summit, attended by thousands, features influential speakers from various sectors, including tech, politics, and activism, aiming to foster diversity, equity, and inclusion. Notable sessions included interviews with YouTube CEO Susan Wojcicki and businesswoman Laurene Powell Jobs, alongside talks by prominent figures like Senator Tammy Baldwin and Stacey Abrams, addressing topics from tech diversity to voter suppression. The event also serves as a significant recruiting platform, with numerous top tech companies participating as sponsors. LWT's initiatives extend beyond the summit, offering programs like the Edie Windsor Coding Scholarship and include.io, which connects underrepresented groups with mentors and employers. The author, Tara Marchand, a Senior Growth Engineer at New Relic, reflects on her experiences at the summit, highlighting the vibrant community and the opportunity to engage with other attendees and industry leaders.

Canary deploys are a crucial tool for modern software teams to roll out new changes reliably and minimize customer impact when they go wrong. At New Relic, we believe canaries provide the most effective means of deploying new services with minimal disruption. Effective canary deployments require finding the right balance between three concerns: canary time, canary size, and metric selection. By following nine best practices, including always using canaries with deploys, timing canary deploys with traffic cycles, and evaluating canaries on core metrics like error rates and response times, teams can ensure successful canary deployments that minimize disruption to users and infrastructure. These best practices make sense for organizations of various scales and complexities, but it's essential to tailor them to individual needs and environments.

In the realm of software deployment, canary deploys are highlighted as an effective strategy for introducing new changes with minimal disruption, allowing for quick rollback if issues arise. New Relic advocates for canary deploys, which involve releasing new functionalities to a subset of servers to monitor performance before a full rollout. The blog outlines best practices for canary deployments, emphasizing the importance of careful planning, defining appropriate metrics, and ensuring the canary process covers a significant portion of the workload without compromising service stability. The balance between canary time, size, and metric selection is crucial, as highlighted by Google engineer Štěpán Davidovič's "triangle of canarying," which guides these deployments to enhance predictive accuracy while minimizing risks. The piece underscores that while canary deploys are powerful tools for maintaining reliability and speed in software updates, their success depends on thoughtful design tailored to the specific needs and scale of an organization.