September 2015 Summaries

Building an effective team in operations requires more than just selecting skilled individuals; it involves understanding each member's capabilities and workload, which is where PagerDuty's new User Reporting tool comes into play. This feature, part of the Advanced Analytics suite, allows managers to track how team members respond to incidents, offering insights into metrics like incident acknowledgment, reassignment, and escalation. These metrics help managers ensure that team members are appropriately positioned and workloads are balanced. Rather than viewing escalations as signs of laziness, User Reporting encourages a deeper analysis of factors such as workload and incident severity that may influence response behaviors. This tool provides a clearer picture of bottlenecks in incident response, enabling techniques like blameless post-mortems and the Five Whys to identify root causes and improve processes. Ultimately, by optimizing response times, teams can focus more on preventive strategies rather than merely reacting to issues, enhancing overall efficiency in incident management. User Reporting is available with PagerDuty’s Standard and Enterprise plans.

StatusCast, led by CEO Alex Bloom, is an application designed to improve customer satisfaction and loyalty by providing real-time updates on service uptime through a hosted status page that can be quickly set up to communicate with users via various channels like email, SMS, and Slack. The tool integrates seamlessly with PagerDuty, an operations management platform, to efficiently send out status updates triggered by alerts, using a webhook integration to choose which alerts to forward, how to phrase them, and to whom they should be sent, based on user preferences and impacted components. This integration allows businesses to maintain transparency and build trust with their customers by notifying them of scheduled maintenance, unplanned disruptions, and other important updates, ultimately enhancing both customer experience and IT operations.

The text explores the distinction between "critical" and "urgent" in incident response, particularly in the context of PagerDuty's incident management. It discusses how a staging environment may be critical for business operations but not necessarily urgent at all times, such as during off-hours, emphasizing the importance of distinguishing between these terms to avoid unnecessary stress and burnout for on-call engineers. The piece highlights the benefits of having a range of alert responses to provide early warnings of potential issues while allowing users to prioritize incidents based on urgency rather than just criticality. PagerDuty offers features like Incident Urgencies to help users manage alerts more effectively, ensuring that only truly urgent issues result in immediate notifications, thereby improving on-call experiences and reducing unnecessary disruptions.

PagerDuty announced its first custom alert sound contest, inviting its creative community to submit unique sounds for potential inclusion in the company's mobile app. The contest, which began on September 21, 2015, encouraged participants to submit a variety of sounds, including songs, clever noises, and avant-garde recordings, with submissions closing on October 2. Voting took place from October 5-8, and the winner was announced at the AWS re:Invent conference in October, with their sound being featured in the app's November release. The winner also received a Jambox, while voters were entered into a drawing for a chance to win one of two custom Jamboxes. Participants were reminded to submit original works no longer than 30 seconds, with an emphasis on short and distinctive sounds.

PagerDuty has introduced Incident Urgencies to help users manage alert fatigue by allowing them to categorize incidents based on urgency, ensuring only critical alerts disrupt on-call engineers during off-hours. This feature enables users to sort incidents into high or low urgency, with customized notification rules that prevent non-critical issues from escalating unnecessarily, thus maintaining a comprehensive view of system health without sacrificing quality of life. By differentiating between urgent and non-urgent issues and allowing for time-based adjustments, PagerDuty ensures teams are better prepared to handle incidents by providing more complete analytics and insight into potential problems. The new snooze functionality also aids in managing non-critical events by allowing teams to pause alerts that do not require immediate resolution, enhancing overall incident management and prevention strategies.

Companies often underestimate the importance of maintaining the morale of their engineers, whose performance is crucial to the success of their products. Burnout among engineers is costly, as it can lead to reduced productivity and high turnover, with replacement costs reaching up to 150% of an employee's annual salary. Maintaining high morale is vital not only for reducing these costs but also for fostering creativity and initiative, which are essential for effective software development. The hero culture and disorganized schedules prevalent in many organizations contribute to stress and undetected issues in software, highlighting the need for structured incident management solutions. These solutions can help by organizing on-call schedules, reducing alert fatigue, improving visibility across teams, and setting realistic expectations, ultimately enhancing collaboration and morale. Investing in these measures requires proactive engagement from IT managers and executives, as well as open communication with engineers, to cultivate a supportive environment aligned with DevOps principles.