June 2014 Summaries

Arup Chakrabarti, operations engineer manager at PagerDuty, delivered a talk at Heavybit Industries discussing common pitfalls in operations teams and strategies to prevent them. Key issues include improper infrastructure setup, where personal accounts are often used instead of corporate ones, and the importance of selecting appropriate tools for deployments. Chakrabarti also emphasizes the need for separate test and production environments, effective configuration management with tools like Ansible, and consistent deployment practices using automation and orchestration tools. Incident management processes should be well-documented and inclusive of everyone involved in production code, necessitating platforms like PagerDuty for effective communication. Monitoring and alerting systems are essential from the start, with tools such as StatsD and Datadog recommended. Regular backups to platforms like S3 and adherence to high availability principles, including using multiple servers and clustered datastores, are vital. Security practices are highlighted, focusing on using gateway boxes instead of SSH, configuring individual user accounts, and activating encryption during development. Finally, internal IT needs, such as consistent hardware and efficient onboarding processes, are important for operational efficiency.

Outages can significantly impact customer trust, but by providing transparent and timely updates, companies can mitigate backlash and build confidence. PagerDuty's integration with StatusPage.io enables seamless communication of system performance to both internal teams and customers. This integration allows incidents to be automatically linked to status pages, ensuring all relevant parties are informed without overwhelming engineers with emails, thus allowing them to focus on resolving issues. For instance, Zumba Fitness has found that using StatusPage.io streamlines communication, reducing workplace disruption during outages. Customers can subscribe to specific notifications, turning potential negative experiences into opportunities for memorable service interactions. Additionally, the integration allows for detailed performance updates, such as uptime and response time, enhancing customer autonomy in diagnosing system issues. By facilitating direct communication channels without the need for manual messaging, companies can maintain operational transparency and reliability.

Rainforest Connection (RFCx), founded by software developer Topher White, is tackling illegal logging and poaching through an innovative use of recycled cell phones transformed into solar-powered listening devices that detect chainsaw activity in real time. This system, initially inspired by White's firsthand experiences with illegal logging in Indonesia, aims to provide immediate alerts to rangers, allowing them to intervene and prevent deforestation before significant damage occurs. Utilizing sensitive microphones, cloud-based APIs, and the alert system PagerDuty, these devices can identify unique sound signatures, such as chainsaws, and notify rangers via various communication methods, facilitating timely responses. Despite the logistical challenges of operating in remote rainforests, RFCx has developed specialized solar panels to keep the system powered and operational around the clock. The organization's efforts not only focus on stopping illegal logging but also aim to engage global audiences by allowing them to listen to the sounds of the rainforest, potentially fostering more active participation in conservation efforts. Looking ahead, RFCx envisions expanding its technology to monitor animal distress and other environmental threats, thereby broadening its impact on conservation and biodiversity protection.

PagerDuty's security management strategy emphasizes high availability and reliability through a system of centralized policy management and distributed enforcement, reducing single points of failure and enhancing fault tolerance. Their approach includes dynamic local firewalls and point-to-point encryption based on IPSec, which allows for efficient, scalable, and secure communication between nodes without relying on vulnerable VPN gateways. By migrating to a Service Oriented Architecture, PagerDuty isolates services to prevent lateral movements and ensures secure operations even if individual servers fail. Furthermore, their role-based access control system, implemented with Chef and Linux user groups, supports a least-privilege permissions model, facilitating secure and efficient access management via version-controlled JSON configurations.

PagerDuty experienced two significant SEV-1 outages on June 3rd and 4th, impacting their Notification Pipeline due to issues with their Cassandra NoSQL datastore. The first outage on June 3rd resulted in delayed notifications and degraded performance, while the more severe outage on June 4th led to a substantial portion of events and notifications being delayed or undelivered. The outages were linked to the background repair process of Cassandra, which strained the system when combined with high workloads, causing instability. Efforts to stabilize the situation initially included stopping the repair process and reducing the load, but these measures were insufficient, leading to a drastic "factory reset" to regain control. PagerDuty acknowledged being underscaled and sharing the cluster among various services with different load patterns, which contributed to the problem. They are implementing changes to prevent future outages, including scaling up Cassandra nodes, setting up multiple clusters, and bringing in additional expertise, while also acknowledging that some planned improvements were delayed due to prioritization based on efficiency.

PagerDuty prioritizes customer satisfaction by implementing a set of unique tools and processes to address support issues efficiently. To maintain a swift response time, they developed a dashboard that visually tracks open tickets, enabling agents to manage their workload and ensure timely responses within a two-hour window. In instances where this internal service level agreement (SLA) isn't met, a PagerDuty incident is triggered, escalating the issue among team members via HipChat, ensuring collaborative problem-solving. Their commitment to customer satisfaction is reflected in their high ratings, with monthly scores consistently near 100%, facilitated by Zendesk's automated emails and GoodData analytics. Additionally, tools like Olark, JIRA, and Skitch are utilized to streamline communication and support, underscoring PagerDuty's dedication to transparent and effective customer service.

PagerDuty emphasizes the importance of security in protecting customer data and ensuring secure communication, given their focus on high availability and the challenges of maintaining robust security in a dynamic, distributed, and fault-tolerant environment. The company employs several best practices, including establishing flexible internal security standards, securing systems by default to enforce consistency, and adopting a provider-agnostic approach to avoid vendor lock-in. They utilize commonly available Linux tools and Chef for security tooling, centralize policy management while distributing enforcement across network nodes, and constantly validate their security measures through active monitoring and alerting to quickly address issues. This blog post is the first in a series that delves into the strategies and techniques used by PagerDuty’s Operations Engineering team to secure their platform.

The latest update to the iOS and Android mobile apps introduces the ability to set custom sounds for push notifications, a feature that has been highly requested by users who found standard notification sounds easily overlooked. With this update, users can choose from 10 different sound options to help distinguish PagerDuty alerts from other notifications, enhancing the likelihood of noticing important alerts. Android users benefit from an additional feature where sounds can play even if the phone is on silent. Push notifications facilitate quick responses, allowing users to view incident details and take action directly from the notification. While push notifications are effective, the developers recommend a backup notification method, such as phone calls or SMS, to ensure alerts aren't missed. The app is available for download on both iOS and Android, and users are encouraged to provide feedback or suggest new features via email or Twitter.