October 2013 Summaries

On-call scheduling has largely remained unchanged since 2011, with many teams initially relying on email alerts to the entire team, leading to unequal incident management and constant on-call status for everyone. As teams expand, a structured on-call rotation becomes necessary, yet challenging to implement due to factors like geographical dispersion and individual alert preferences. Various on-call methods are used today, each with its own drawbacks: dedicated phones or pagers can lead to unfair burdens, network operations centers may delay response times, email blasts create alert fatigue, and automated systems often miss escalation scenarios. Additionally, limited flexibility in native scheduling tools and the complexity of managing multiple monitoring tools are common issues. The text suggests that teams experiencing these challenges should consider a comprehensive incident management solution, such as PagerDuty, to improve on-call scheduling and maintain team well-being.

PagerDuty has introduced several new features aimed at enhancing incident management and response efficiency. The Dial Extensions feature simplifies receiving alerts on office phones by navigating complex phone trees through the use of commas for dialing delays, thus improving accessibility. Users can now add descriptions to schedules, services, escalation policies, and user profiles, facilitating better organization and context management, although these fields are limited to 1024 characters. Additionally, URLs within incidents are now clickable, allowing for easier access to supplementary information without the need for manual copying and pasting. Furthermore, the integration of JavaScript through CORS is now live, enabling users to trigger, acknowledge, and resolve alerts directly from their browsers, showcasing a significant step towards improved user interaction and responsiveness. The team encourages feedback from users who have experienced these updates.

PagerDuty has launched a revamped mobile app for iOS and Android, enhancing incident management capabilities on-the-go by allowing teams to respond quickly to IT issues through phone call, SMS, and push notifications worldwide. The app's redesigned interface offers a Global Status Indicator for easy monitoring and allows users to acknowledge, resolve, or re-assign incidents with simple swipes and taps, addressing previous usability concerns. Feedback from early adopters during the app's beta phase significantly influenced its development, ensuring it meets user needs and improves response times. Available for free on Apple’s App Store and Google Play, the app aims to offer seamless incident management anytime, anywhere, supported by continuous updates based on customer input.

At the Nagios World Conference North America, Arup Chakrabarti from PagerDuty discussed effective alert management in production systems, emphasizing the importance of filtering out non-actionable alerts to focus on those impacting customer experience. The proliferation of alerting due to cheaper computing and automation can lead to alert fatigue, making it crucial to discern which alerts truly matter, particularly those related to customer-facing issues like website availability for e-commerce. Chakrabarti suggests analyzing alert history to evaluate incident severity and implementing a tiered alert system, such as tagging alerts with severity levels, to ensure only critical issues disturb engineers during off-hours. By managing alerts effectively, organizations can maintain a sense of urgency for significant problems and reduce the mean time to resolution, allowing engineers to rest assured that non-critical alerts will be addressed during regular hours, thus combating alert fatigue.

At PagerDuty, a monthly Hack Day encourages creativity and innovation among employees, leading to the development of unique projects that enhance productivity and team spirit. During one Hack Day, Product Manager David Hayes created an integration between PagerDuty and his bedroom lights using a Belkin WeMo switch and If This Then That, solving his issue of waking up for alerts during deep sleep. Similarly, software engineer Cory Chamblin developed an integration between PagerDuty and SnapChat to receive alerts in a personalized and engaging manner, despite the absence of a public API for SnapChat. These inventive solutions reflect the dynamic and inventive spirit fostered during Hack Days at PagerDuty's San Francisco and Toronto offices, with anticipation for further creative projects in future sessions.

New Relic is hosting its first user conference, FutureStack, where they are introducing new mobile and social analytics, along with app monitoring features. A significant highlight for both New Relic and PagerDuty customers is the launch of Alert Policies, which enhances alert notifications by enabling users to set thresholds, share alert conditions across various platforms, and direct alerts to the appropriate individuals or teams. This new feature helps reduce unnecessary alerts and allows customers to focus on critical notifications by configuring Notification Channels and creating specific notification groups. New Relic customers can integrate PagerDuty as an alert channel, leveraging its customizable on-call scheduling and escalation policies to ensure timely and relevant responses. The integration guide provides details on setting up PagerDuty as an alert channel, and feedback from users on the new Alert Policies is encouraged.

PagerDuty has announced a new integration with CopperEgg, a SaaS-based server monitoring service, to enhance communication and streamline data synchronization between the two platforms. This integration allows PagerDuty to directly interact with CopperEgg, providing efficient on-call scheduling, alerting, and incident tracking, which helps teams respond swiftly to server issues and prevent outages. Both DirectWest and Eye-Fi have reported benefits from the integration, as it offers a simplified and consolidated alert system that ensures seamless service operation and customer access. This collaboration exemplifies how API integrations can expand the functionality of monitoring tools beyond traditional email and text notifications, and users are encouraged to explore the integration guide for setup.

AWS CloudWatch now integrates with PagerDuty, enhancing the capabilities of both platforms to monitor and manage IT environments effectively. The integration addresses the needs of the 13% of PagerDuty users who already deploy AWS CloudWatch, facilitating seamless data exchange and incident management. AWS CloudWatch offers extensive resources and monitoring solutions, tracking system metrics to ensure smooth operations and providing detailed statistics on system issues. When CloudWatch alarms are triggered, PagerDuty alerts users with comprehensive details, streamlining the diagnosis and resolution process without needing to access multiple services. Users can customize notification thresholds for over 100 metrics, ensuring timely alerts and efficient incident resolution. The integration operates via a simple notification service (SNS) topic that links CloudWatch data to PagerDuty, enabling rapid communication and action from on-call teams.

Rollbar is a tool that monitors application errors at the code level, helping developers identify and resolve issues more efficiently by aggregating similar problems and revealing patterns. It has integrated with PagerDuty to enhance its service by offering alerting, on-call scheduling, and incident management capabilities, allowing customers to create, update, and resolve incidents seamlessly. Richard Lee, CTO of the mobile development studio Polydice, highlights the importance of this integration in ensuring continuous functionality for their iOS apps by receiving immediate notifications of errors to prevent downtime. The integration process is straightforward, involving the addition of a service and selecting the Generic API System in PagerDuty, followed by enabling notifications in Rollbar.

Event Enrichment has developed a methodology to reduce noise from IT events and decrease the mean time to repair incidents, with a focus on customizing Nagios events sent to PagerDuty. This customization helps operations teams by filtering out unnecessary environmental variables, making incidents more manageable and reducing disruptions such as being unnecessarily alerted in the middle of the night. Recognizing the popularity of PagerDuty, especially among the over 40% of users who also have Nagios deployed, Event Enrichment provides detailed guides and educational materials to enhance incident management through this integration. By promoting their methodology globally, they aim to improve the efficiency and effectiveness of Network Management Systems, as highlighted by Ophir Ronen from EventEnrichment.org.

PagerDuty has announced its integration with Zapier, enabling users to exchange data between PagerDuty and over 242 other apps that integrate with Zapier. This integration allows for automation of tasks such as creating JIRA tickets from PagerDuty alerts, enhancing team organization and incident management. Additionally, the integration transforms existing one-way integrations, like with Zendesk, into bidirectional ones, allowing users to create Zendesk tickets using customer information. Users can also create custom integrations with various services like Basecamp, Gmail, and GitHub by designing personalized workflows through Zapier's platform. The integration aims to streamline processes and encourage creativity in workflow design, inviting users to share their integration plans in the comments.