February 2014 Summaries

Incident Volume and Mean Time to Repair (MTTR) are crucial metrics for on-call teams to track in order to identify systemic issues and improve the speed of resolving incidents. The process involves using Zoho Reports to pull data from the PagerDuty API, allowing teams to create dashboards that display incident volumes and MTTR across different services and escalation policies. By setting up automated data imports and using SQL queries, teams can generate visual reports to analyze incident trends and identify areas for improvement. The guide provides detailed steps for setting up Zoho Reports to graphically represent data, enabling teams to filter by variables like date and escalation policy, and ultimately helping team leads to focus on services they are responsible for. Sharing these insights with the organization is made easy through Zoho's sharing features, which allow embedding reports in internal platforms like Confluence, fostering better communication and collaboration within the company.

In this blog post, Shawn Parrish from NodePing shares insights and lessons learned from common server monitoring mistakes, highlighting the importance of comprehensive and frequent monitoring to ensure server reliability and performance. He emphasizes the necessity of checking all servers, including secondary, new, cloud, and temporary servers, as well as all services on a host to prevent unnoticed failures. Parrish advises configuring monitoring intervals to one minute for timely alerts and stresses the significance of HTTP content checks, proper timeout settings, and reverse DNS monitoring to avoid service disruptions. He also discusses the need for appropriate alert sensitivity, correct notification settings, and the importance of selecting cost-effective monitoring services. By addressing these common pitfalls, Parrish aims to help others maintain robust server monitoring practices without unnecessary expenses.

PC Monitor, a trusted monitoring system used by over 250,000 consumers and professionals worldwide, offers a seamless integration with PagerDuty to help IT administrators effectively manage and respond to system issues remotely via smartphones or tablets. With the rise of smartphone use and the need for IT systems to run smoothly around the clock, the combination of PC Monitor and PagerDuty allows users to set up alerts and take immediate action from their mobile devices. IT administrators can utilize this system to execute Terminal commands, restart, or shut down systems directly through the app, ensuring quick resolutions to any problems. The collaboration between these platforms emphasizes the importance of having a well-prepared plan of attack and ensuring multiple team members are informed and ready to respond to alerts, thus enhancing the efficiency and reliability of IT management in a connected world.

PagerDuty emphasizes the value of customer feedback, which has inspired collaborative updates with MongoDB to enhance monitoring and alerting capabilities for MongoDB clusters. MongoDB responded to customer demands for increased visibility and proactive issue management by creating the MongoDB Management Service (MMS), which provides free monitoring and premium backup options. By integrating MMS with PagerDuty, users gain the advantage of smart alert routing, flexible on-call scheduling, and automatic escalations, ensuring that issues are promptly addressed by the appropriate team member. This partnership allows users to resolve problems swiftly and efficiently, maintaining operational stability and preventing potential disasters. MMS Monitoring, a complimentary service, offers users insights into critical server and hardware indicators, and when combined with PagerDuty, facilitates quick issue detection and management across a customer's infrastructure.

John Sheehan, CEO of Runscope, highlights the critical importance of robust API monitoring in modern, distributed applications, where APIs serve as the connective tissue between various components. Traditional HTTP monitoring tools effectively address questions of availability and performance by measuring response times and uptime through frequent endpoint checks. However, ensuring API correctness requires a more nuanced approach that considers read and write operations and complex workflows involving multiple sequential API calls. Runscope Radar addresses these needs with a flexible assertion framework that verifies API responses against expected outcomes, enabling the extraction and chaining of data across requests. Integrations with platforms like GitHub, Zapier, and CircleCI facilitate seamless inclusion of API monitoring and testing in existing deployment processes, while connectivity with PagerDuty allows for immediate alerts in case of test failures. Runscope offers interested PagerDuty customers the opportunity to try its service with a complimentary Standard account.

PagerDuty offers a comprehensive solution for managing on-call schedules and alert routing to minimize the stress associated with being on-call. The platform enables flexible, rotating schedules that accommodate global teams through features like Follow-the-Sun rotations and customizable shift layers, ensuring fair distribution of responsibilities without disrupting personal time. Hand-off notifications and schedule exports facilitate seamless transitions and help users keep track of their shifts. To prevent incidents from being missed, PagerDuty employs Escalation Policies that automatically reroute alerts to secondary on-call engineers or managers if necessary. This ensures timely responses while allowing primary on-call engineers to rely on backups when needed. By leveraging these tools, PagerDuty aims to provide users with a balanced approach to on-call duties, allowing them to maintain a healthy work-life balance.

The competition invites DevOps professionals to share personal stories of on-call heroism on various online platforms, with the aim of gaining internet fame and a chance to win prizes. Participants are encouraged to recount their most challenging and memorable on-call experiences, detailing their reactions and solutions to the problems encountered. Submissions must be publicly accessible and shared on Twitter with the hashtag #OnPagerDuty by February 28, 2014, to be eligible for prizes, including a Parrot AR Drone and a Horse Mask for the grand prize, among others. Winners will be announced in mid-March, and no purchase is necessary to enter the contest.

CopperEgg, a cloud application monitoring partner, offers insights into creating an effective alerting process by analyzing historical performance data. The post outlines a structured approach to optimize performance by first reviewing historical trends and metrics like response time and availability, especially during high-traffic periods. CopperEgg provides high-resolution performance updates, enabling users to analyze and drill down into specific issues. The next step involves identifying root causes of performance problems using CopperEgg's tools, which allow for detailed insights with minimal navigation. The process continues with setting performance goals based on business needs and past performance, followed by creating specific alerts to anticipate potential issues. CopperEgg allows users to set customized alert thresholds, and integration with PagerDuty ensures that alerts are efficiently routed and escalated to the right personnel. The combination of CopperEgg and PagerDuty aims to provide a comprehensive monitoring solution, ensuring timely alerts and effective incident management, with a free 14-day trial available for new users.

When on-call, it's important to remain accessible to technology and avoid activities that could hinder responsiveness, such as embarking on remote hikes without connectivity or attending events like concerts where alerts might be drowned out. Staying away from places that require phones to be turned off, like cinemas, is advisable, as is avoiding excessive alcohol consumption that could lead to embarrassing mistakes. While it's encouraged to leave the office and enjoy life, it's essential to be prepared for any work emergencies by keeping necessary tools like a laptop handy. The text humorously emphasizes maintaining a balance between being available for on-call duties and living a normal life without compromising professional responsibilities.