August 2014 Summaries

Daniel Gentleman created a PagerDuty Arduino integration using an Arduino YUN, a 3D printed sphere, and Adafruit’s Neopixel Ring to provide a visual status update of his PagerDuty dashboard. By utilizing PagerDuty’s API, the device checks for alerts every 15 seconds and changes color accordingly: green for resolved alerts, orange for acknowledged alerts, and red for triggered alerts. To ensure reliability, Daniel implemented a WiFi connection check, pinging Bing.com every second, and the lamp fades to blue if the connection is lost. The simple shell scripts and single I/O pin usage allow for potential future expansions, such as indicating who is on-call or integrating with a robot.

Dave Josephsen, a developer evangelist at Librato, discusses the complexities and challenges of monitoring systems, emphasizing the need for a unified approach to signal processing to ensure reliable feedback from systems that cannot be fully controlled. He highlights the common problem of disparate monitoring systems within organizations, where different teams use separate tools that lead to redundant and sometimes conflicting signals, causing confusion and inefficiencies. Josephsen suggests that the solution lies in synchronizing alerting and visualization on a common input signal, which can be achieved without requiring everyone in the organization to use the same tool, by ensuring that processing and notification systems use a common data stream. He recommends using platforms like Librato and PagerDuty to centralize telemetry signals, facilitating effective event correlation, visualization, and notification, thus allowing teams to maintain their preferred tools while aligning on a unified signal for consistent feedback.

PagerDuty's Advanced Reporting feature assists teams in monitoring and improving their operational performance by focusing on key metrics such as Raw Incident Count, Mean Time to Resolution (MTTR), Time to Acknowledgement, and Escalations. It emphasizes the importance of understanding incident spikes to identify infrastructure issues or misconfigured monitoring tools, and highlights the necessity of quick resolution times to maintain revenue and customer loyalty, as illustrated by Major League Soccer's focus on minimizing downtime. The feature also underscores the value of tracking Time to Acknowledgement to gauge team readiness, with operationally mature teams maintaining high response expectations and utilizing escalation timeouts to enforce response targets. While escalations can indicate gaps in response capabilities, they are sometimes part of standard procedures, and understanding the types of alerts that warrant escalation helps teams optimize their incident management processes.

Evan Gilman, an operations engineer at PagerDuty, emphasized the critical importance of robust security practices during a recent talk at a meetup held at PagerDuty HQ. He highlighted the challenges of maintaining security in both small and large organizations, stressing the need for a scalable approach to fend off persistent attackers. Emphasizing a "secure by default" mindset, Evan advocated for stringent security checks and the protection of all data, even logs, by assuming a constantly hostile network environment, particularly in cloud settings. He recommended encrypting all network traffic at the transport layer and sanitizing outbound data to protect against provider vulnerabilities. Automation plays a key role in Gilman's strategy, suggesting the distribution of security policies through a centralized ruleset that individual nodes can enforce locally, exemplified by role-based access controls. Monitoring and responsive alert systems are crucial, as evidenced by the need to track encryption levels and the use of host-based intrusion detection systems (HIDS) due to the absence of virtual private clouds (VPC) in PagerDuty's distributed infrastructure. Concluding with a call to action, Gilman urged the audience to begin implementing security measures immediately to mitigate technical debt and address existing vulnerabilities.

PagerDuty offers a comprehensive solution to common engineering challenges related to infrastructure monitoring, alert management, and on-call shift changes, which can otherwise be costly and complex to address independently. The platform consolidates multiple monitoring tools, streamlines the process of sending and rerouting alerts, simplifies on-call shift transitions, and provides dashboards for identifying potential issues proactively. By using PagerDuty, businesses can avoid the significant expense and effort required to develop their own tools, as the company has dedicated over five years to perfecting these solutions.

Simon Westlake, the Chief Technical Officer of Powercode, a CRM, OSS, and billing system for ISPs, discusses their integration with PagerDuty, which has been highly beneficial for their global clientele. Despite Powercode's numerous integrations with third-party services, only a few are widely utilized, with PagerDuty being a notable exception. The decision to integrate PagerDuty was prompted by the need to manage after-hours emergencies effectively, as smaller ISPs often lack the resources for a 24/7 network operations center. The integration was seamless due to PagerDuty's well-documented and developer-friendly API, which aligned with Powercode's strict integration requirements. The process involved decisions about alert configurations and avoiding two-way communication to maintain Powercode as the primary incident management interface, leading to efficient incident tracking and resolution within PagerDuty. The integration utilizes features like incident keys and real-time data access, and its success has led to its use in Powercode's own after-hours support, underscoring its value to both Powercode and its customers.

Stephen Lee, director of platform solutions at Okta, highlighted the importance of secure access control during a PagerDuty security meetup, emphasizing the role of automation in provisioning applications for new employees to enhance productivity and reduce IT workload. He addressed the challenges posed by the increasing use of mobile devices in workplaces, which complicates data security due to the potential loss or theft of personal devices containing company data. With the widespread adoption of cloud technology, enterprises face challenges in managing authentication and authorization across various locations and devices, necessitating robust systems for access control and secure connections. Stephen advocated for a security approach focused on "identity lifecycles," which involves mapping out access control for both internal and external users and integrating security checks into the product development process. This lifecycle approach requires a holistic view of security and considers the end-users' needs, positioning them as "customers" of the security team to ensure they have access to necessary resources without compromising security.