November 2013 Summaries

PagerDuty's recent Hack Day showcased a variety of innovative projects, highlighting the company's creative talent. Among the featured projects was the PagerDuty Clapper, created by an intern using a Raspberry Pi to integrate with PagerDuty via webhooks, allowing incidents to be acknowledged or resolved with claps. Another project, the 3D Incident Visualizer, developed by software engineers to map server locations for real-time incident tracking, used a modified WebGL globe for visualization. Additionally, a video celebrating PagerDuty's on-call team provided an inside look into the lives of those who ensure high availability for customers, emphasizing both their dedication and their relatable, everyday interests outside of work.

PagerDuty's Internal Tools and Operations team plays a crucial role in ensuring the seamless functioning of the company's internal applications, analyzing trends, and forecasting growth. Key team members include Kyle Napierkowski, the Data Guru, who transforms customer data into actionable insights while embracing a diverse skill set akin to a storyteller and scientist. Joel Eidsath, a Data Engineer, is known for his data management and values the friendships formed within the company. Eric Walker, the newest team member, aids the IT group in developing a data warehouse to synthesize and analyze data from various sources. Baskar Puvanathasan, a co-founder, has dedicated significant efforts to building PagerDuty into a leading IT entity, balancing his professional achievements with personal hobbies.

During AWS re:Invent, a team collaborated with StackDriver to conduct a survey on IT environments, offering participants poker chips for their insights. The survey revealed significant dissatisfaction with current monitoring and alerting tools, as 64% of respondents expressed dissatisfaction with their monitoring tools, and only 31% were happy with their alerting methods. It was noted that centralized alerting tools, like PagerDuty, were underutilized, with only 34% of attendees reporting their use, while others relied on less efficient methods such as homegrown SMS gateways and email notifications. The event highlighted the ongoing challenges in IT environments and the need for improved solutions, with gratitude extended to those who participated in the survey.

PagerDuty places a strong emphasis on reliability, deploying its code across multiple data centers and cloud providers to ensure uninterrupted alert services via phone, SMS, push notifications, and email. Despite their failure-tolerant design, implementation issues can arise, prompting the need for proactive strategies like the "Failure Friday" exercises. These weekly events involve intentionally introducing failures into their systems to uncover weaknesses, improve resilience, and enhance team collaboration. During these exercises, the team conducts a series of tests, such as stopping services, rebooting hosts, and simulating network issues, to evaluate and improve their systems' robustness. Communication is crucial during these sessions, with dedicated chat rooms and conference calls facilitating quick information exchange and logging actions for review. The insights gained are used to prevent future outages and reinforce PagerDuty's commitment to maintaining its high standards of reliability.

Blackrock 3 Partners LLC draws parallels between the emergency response operations of fire departments and the incident management processes of DevOps teams, emphasizing the shift from "Peacetime" to "Wartime" when a severe incident occurs. During Peacetime, normal operations proceed, but Wartime demands rapid, decisive action, where communication becomes direct and concise, akin to "Voice Twitter." The text describes how incident management in both fields relies on structured roles, clear communication, and specialized resources to resolve crises effectively. It highlights that just as fire departments have honed their emergency response over decades, DevOps teams must adopt a similar mentality and process to tackle high-severity incidents, ensuring robust communication systems to coordinate their response efficiently. The blog underscores the importance of having a defined Wartime organizational structure and communication strategy for timely and effective incident resolution.

PagerDuty's mobile app, initially a simple mobile view of their web app with push notifications, faced criticism for its sluggish performance and lack of optimization, prompting a complete redesign focused on speed and user experience. The original app used jQuery Mobile, which led to slow performance and a complex navigation structure, resulting in customer dissatisfaction. To address these issues, the development team shifted to a more efficient set of libraries, including Hammer.js for touch gesture support and Zepto for improved performance, while emphasizing a streamlined user interface that surfaces essential information quickly. By engaging users in a public beta and incorporating their feedback, PagerDuty significantly enhanced the app's usability, leading to increased satisfaction and higher ratings in app stores, as evidenced by positive user testimonials.

PagerDuty's Support Team plays a crucial role in maintaining customer satisfaction by educating users, handling feature requests, and resolving issues promptly. Led by Support Team Manager Ryan Hoskin, who oversees the allocation and prioritization of tasks, the team is committed to exceeding customer expectations. Technical Support Engineers like Julian Martinez and Dan Kheronsky contribute by explaining product functionalities, routing bugs, and crafting integration guides to enhance user experience. The team's passion extends beyond their roles, with personal interests such as hiking, biking, woodworking, and enjoying drinks with historical figures, reflecting a well-rounded and dedicated professional environment.

PagerDuty utilizes Chef for automating its computing infrastructure and emphasizes the importance of testing Chef code before deployment to maintain a stable environment. The testing process involves several sequential strategies, including semantic testing with Foodcritic to catch mistakes early, unit testing with ChefSpec for robust coverage of recipes, and functional testing that simulates real production environments using Linux containers. Integration testing is conducted in two main environments to ensure code compatibility with application features, while "Failure Fridays" are used to intentionally inject failures for resilience testing. These automated testing methods enable PagerDuty to continuously deliver high-quality code, reducing time to market and ensuring a dependable infrastructure for its customers.

PagerDuty has announced new integrations with AppFirst, Pulseway, Lyatiss CloudWeaver, and Slack to enhance its alerting and incident management capabilities, offering seamless interaction with existing monitoring tools and business software. The integration with AppFirst aims to provide IT operational insights by correlating metrics at the intersection of applications and infrastructure, ensuring better preparedness for handling issues. Pulseway's integration is designed to add value to their monitoring solutions by incorporating alerting and incident tracking, making it the first source of notification for system issues. Lyatiss CloudWeaver integration is deemed essential by DevOps professionals for centralized alert management, reducing the number of systems monitored. Finally, the integration with Slack aims to consolidate team communication and ensure reliability in IT systems by providing real-time notifications and automatic escalation, which Slack already utilizes to maintain business operations and customer satisfaction. Each integration aims to offer users enhanced control and efficiency in managing critical IT operations.

Attendees of Amazon re:Invent in Las Vegas can visit PagerDuty's booth to explore their latest offerings and participate in a joint survey with Stackdriver, which rewards participants with poker chips for casino play. PagerDuty and Stackdriver are also co-sponsoring a Pub Crawl at AquaKnox in the Venetian, featuring an open bar and lively atmosphere, starting at 5:30 PM on Wednesday, with early entry available through a pre-Happy Hour at StackDriver's booth. Those unable to attend can follow updates and survey results via PagerDuty's Twitter handle and the #AWSreinvent hashtag.

The On-Call Scheduling Best Practices Series emphasizes the importance of proactive planning and adherence to specific strategies to efficiently manage IT outages. Key recommendations include maintaining a consistent and predictable schedule to avoid surprises, appointing a single individual to oversee scheduling with team input, and ensuring that the system is automated for ease of maintenance. Additionally, the series advises having a robust alert policy to manage escalations and involving more personnel if necessary, as well as regularly reviewing incident management metrics to improve response times and reduce alert fatigue. It also highlights the significance of fostering a customer-centric culture, ensuring that the on-call process supports delivering reliable software to customers. Feedback from the team is crucial for refining the on-call system and gaining their support for its implementation.

PagerDuty has announced a new Mobile Incident Management app designed to improve upon its previous mobile app, which was initially a mobile website enhanced with a native wrapper to enable push notifications for iOS and Android devices. The company discovered that push notifications were rapidly gaining popularity, with 47.25% of nearly one million non-email monthly alerts being push notifications by September 2013, just eight months after the app's initial release. iOS adoption outpaced Android initially, but Android started closing the gap in recent months. Push notifications have significantly reduced the Mean Time to Repair (MTTR) for incidents, with Android users resolving incidents 122% faster and iOS users 76% faster compared to non-push methods like email, phone, or SMS. The new app provides users with quick access to incident snapshots, allowing on-call team members to efficiently manage incidents with features that enable acknowledgment, resolution, or reassignment with a single tap.

High-frequency trading, which constitutes half of US securities trading, relies heavily on robust computer systems to automate trades effectively, but failures can lead to significant financial losses, as demonstrated by Knight Capital's $460 million loss in 2012 due to a software deployment error. The incident highlighted the importance of implementing best practices in IT incident management, such as setting financial thresholds to prevent excessive trading beyond capital capacity, creating a responsive people process to ensure timely intervention when alerts are triggered, and learning from previous incidents to improve control measures and prevent future errors. Knight Capital's lack of adequate procedures and supervision during the deployment of a new code led to a failure to address alerts promptly, exacerbating the problem, and their prior $7.5 million loss in 2011 was a missed opportunity to enhance their systems' resilience and prevent the much larger loss the following year. The cyclical process of setting thresholds, involving the right personnel, and applying lessons from past incidents is essential to maintaining efficient incident management and minimizing financial and reputational damage.

PagerDuty has introduced a new user role called Limited User to enhance user permissions and better manage growing teams. This role allows for more precise control over team members' access to certain functionalities, ensuring that only authorized personnel can make changes to on-call schedules and services. The Limited User role is part of a broader set of user permissions that include Owners, Admins, and Users, each with varying degrees of access, such as updating billing information, managing users, and handling incidents. To assign this role, users can navigate to the Users Tab, input the relevant information, and select "limited user" from the role dropdown menu. Additional information and support are available through PagerDuty's knowledge base and support email.

PagerDuty's Operations Engineering team, comprising four generalist engineers, focuses on infrastructure automation, host-level security, data stores, and productivity tools, and ensures high availability by utilizing resilient software on resilient infrastructure. The team automates its infrastructure using Chef, a ruby-based configuration management tool, and follows a feature branch workflow with unit testing via Jenkins and integration testing in a staging environment. They prioritize using community cookbooks, supplemented by custom wrapper cookbooks with the "pd" prefix, managed by Berkshelf, and employ custom knife plugins for tasks such as server backups and spawns. Despite challenges like cross-component versioning and code quality issues in community cookbooks, the team remains confident in their infrastructure's reliability, thanks to a test-driven development approach and the adoption of security testing automation tools.

PagerDuty has announced a new integration with Honeybadger, aimed at enhancing the monitoring of Ruby applications by providing effective alerting, on-call schedule management, and incident tracking. This integration allows error information to be efficiently routed to the appropriate personnel, thereby preventing unnecessary notifications to those unavailable, such as those on vacation. Although not initially publicized, 5% of Honeybadger's customers have already adopted this integration, and it is expected that more will follow as awareness grows. Additionally, Honeybadger is offering a promotion for PagerDuty customers, providing up to $89 off the first month of their monitoring services, accessible through their website.

PagerDuty is participating in Movember for the third consecutive year, encouraging team members to grow moustaches in support of men's health. The initiative involves starting with a clean shave on November 1, and throughout the month, participants grow their moustaches without connecting them to sideburns or chin hair, as beards and goatees are not permitted. To aid in this effort, PagerDuty's Happiness Team has provided shaving kits to all employees. The event is not just about fun and facial hair but serves as a charity drive to raise funds for the Movember Foundation, with all proceeds from donations going directly to the organization. Participants and supporters are invited to join the PagerDuty Movember team in this fundraising effort, with a follow-up post promised next month.