January 2013 Summaries

PagerDuty has secured $10.7 million in funding, led by Andreessen Horowitz, with contributions from Jesse Robbins, WIN Funding, and existing investors, aiming to accelerate the development of its IT incident tracking and on-call management platform. The company, which has largely relied on word-of-mouth promotion, serves thousands of customers, including major enterprises and startups, by acting as a "9-1-1 dispatch" system for IT engineers to address critical issues. PagerDuty's vision is to evolve into the central nervous system of IT operations, enhancing the signal-to-noise ratio of alerts from various monitoring tools, thereby minimizing false alerts and increasing efficiency. The company is committed to building an intuitive, reliable, and fault-tolerant system that eliminates downtime, deploying distributed data stores like Cassandra and Zookeeper to ensure uninterrupted service. PagerDuty is seeking engineers to join its mission of transforming DevOps tools, emphasizing reliability and user empowerment in resolving incidents efficiently.

PagerDuty has announced the addition of four new team members: David Lanstein, Doug Barth, Ryan Hoskin, and Sam Noland, each bringing unique skills to the company. David, formerly of Splunk, will lead the Sales Department as Director of Major Accounts, leveraging his two years of technical sales experience. Doug joins the operations team, known for his detail-oriented approach and skill in baking, as he works on improving their test suite performance. Ryan, with over eight years of technical customer support expertise, strengthens the support team with his Microsoft proficiency, creating PowerShell scripts to aid Windows users. Sam, joining the Happiness Team, brings her expertise in Vibe Management to enhance company culture across offices, contributing with projects like a Ducksboard status board. These additions reflect PagerDuty’s commitment to growth and reliability.

In January 2013, PagerDuty experienced multiple outages affecting its events API and web application, impacting its customers' ability to submit monitoring events and access accounts. The outages were primarily caused by reliance on a legacy database system for the events API, which required a downtime-inducing failover process when the primary database failed. Immediate responses included flipping to a secondary database and addressing slow queries, but persistent issues led to the decision to upgrade database hardware. Efforts to implement these upgrades were hampered by errors due to engineer fatigue, resulting in further downtime before successfully establishing new database machines. To prevent future incidents, PagerDuty planned short-term measures such as monitoring slow queries, automating database server builds, and improving development processes, alongside long-term plans to re-architect the events API to utilize a more resilient distributed data store capable of enduring data center outages without downtime.

PagerDuty has introduced new features to enhance its mobile site experience, stemming from a recent hackday event. The Mobile On-Call Display feature now allows users to easily check their on-call status and see which team member is on call within any configured Escalation Policy, all from their smartphones. This function, accessible via a phone icon, displays "ON-CALL" or "OFF-CALL" status and the number of incidents assigned, enabling users to acknowledge, escalate, or resolve incidents directly from the mobile site. Additionally, a Mobile User Directory feature has been added, providing a contact button that lists every user in the account with their contact information, facilitating quick communication. Users are encouraged to explore these new mobile functionalities and provide feedback, with the option to participate in future hackday projects.