November 2014 Summaries

Doug Barth, a PagerDuty operations engineer, advocates for eliminating scheduled maintenance to enhance system reliability and operational efficiency. Speaking at DevOps Days Chicago, Barth argues that scheduled maintenance, often occurring late at night or on weekends, is inconvenient for operations staff and limits customer access. He proposes fast, iterative maintenance strategies that avoid system-wide disruptions. Key tactics include deploying in stages with robust, script-based processes, utilizing canary deployments to safely test changes on small system segments, and implementing retries and distribution strategies to ensure system resilience. These approaches, Barth suggests, allow for maintenance during regular hours without compromising service quality, ultimately improving the work-life balance of operations teams and maintaining continuous customer access.

PagerDuty's recent hackday provided an exciting opportunity for employees to explore creative projects and enhance their skills with new technologies, particularly focusing on iOS 8's features. Highlights included Steve's award-winning proof of concept utilizing the Handoff feature for seamless incident response across devices, and Alper and Clay's development of a widget for on-call notifications, both integrated into the newest iOS build. Other notable projects included improvements to developer documentation by Grant, Amanda, and Greg, which involved a new design, enhanced latency and security, and a transition to a Jekyll backend. Additionally, David Y created a bot for checking meeting room availability via HipChat, Chris and the UX team developed an Anki flashcard deck for employee onboarding, Tim wrote a plugin for Lita to access security vulnerabilities, and Shack crafted a guide for Chrome Omnibox searches to assist new employees. The event showcased the company's commitment to innovation and collaboration, inviting others to join future hackdays.

PagerDuty developed a social media policy to foster an inclusive and harassment-free online environment, reflecting its commitment to diversity and innovation. The initiative arose after encountering inappropriate comments on Twitter ads, prompting the company to establish clear guidelines against harassment, irrespective of an individual's identity or affiliations. Despite finding limited examples from other companies, PagerDuty based its policy on a template from the Geek Feminism wiki, ensuring it applies to all its online spaces, including Twitter and Facebook. The policy, though concise at 160 words, is seen as a crucial step in promoting the desired company culture and provides a framework for addressing incidents, typically resolved through private requests for comment removal. The company aims to enhance the policy's visibility among its community while maintaining the integrity of its online interactions.

Mean time to resolution (MTTR) is a crucial metric for operations teams, representing the time between an incident's onset and its resolution, directly impacting system uptime. However, focusing solely on MTTR can be limiting, as overall downtime is influenced by both the frequency and duration of outages. Sustainable improvement in MTTR requires a deep dive into incident response processes, highlighting the importance of how teams collaborate and communicate during outages. Key strategies include refining notification processes to reduce response times, establishing clear protocols and leadership during incidents, and maintaining detailed documentation to aid post-mortem analyses. Regular practice of incident response plans and utilizing instrumentation and analytics to identify issues are also emphasized as essential practices for optimizing resolution times.

SaltStack, an open-source configuration management and remote execution tool, has recently integrated with the PagerDuty platform, enhancing its ability to manage large-scale server infrastructures by monitoring system failures, overseeing infrastructure changes, tracking system vitals, and managing code deployments. This integration allows for real-time alerts and automated processes, ensuring that deployment issues and system anomalies are promptly addressed. SaltStack's functionalities, such as Salt states and monitoring states, work seamlessly with PagerDuty to maintain system compliance and monitor performance thresholds. Users can efficiently manage code deployments by utilizing automated alerts and processes to ensure stability and functionality post-deployment. For more insights into this integration, SaltStack and PagerDuty will be present at AWS re:Invent, offering information and demonstrations at their respective booths.

PagerDuty enthusiastically participates in Movember, an annual event where employees grow mustaches to support the Movember Foundation's efforts in raising awareness for men's health. Both "Mo Bros," who grow and style their mustaches, and "Mo Sistas," who promote the cause and donate, actively contribute to the initiative. Throughout the month, participants enjoy friendly competition, culminating in the crowning of "Mr. Movember" for the best mustache and "Miss Movember" for the most spirited female supporter. The month concludes with a Movember Party where participants celebrate their efforts, shave their mustaches, and capture the final results, all while encouraging others to join their team and track their progress online.