July 2015 Summaries

PagerDuty experienced an unauthorized intrusion on July 9, where an attacker accessed some customer information, but the incident did not impact their service delivery or expose sensitive data like social security or credit card numbers. The company quickly mitigated the breach by enhancing their security measures and engaging a cybersecurity firm to validate their practices, alongside cooperating with law enforcement. The attacker gained access to users' names, email addresses, hashed passwords, and public calendar feed URLs, but due to robust hashing techniques, it is computationally infeasible for the attacker to exploit the passwords. PagerDuty is advising users to reset their passwords and calendar feed URLs as a precaution and warns of potential phishing attacks using the exposed information. The company's leadership has been actively involved in managing the incident, emphasizing their commitment to transparency and security improvement, and users are encouraged to contact PagerDuty support for any concerns.

PagerDuty has rebranded Advanced Analytics to Advanced Reporting, which now includes Team, System, and User Reports, and has introduced PagerDuty Analytics to provide critical operational insights into people, technology, and processes over time. The platform aims to enhance incident management by helping IT operations teams analyze incident trends, reduce non-actionable alerts, and identify common alerts that may indicate larger issues. With improved reporting capabilities, teams can visualize metrics such as common incidents, SLA performance, and noisy incidents, allowing for more granular insights that drive improvements in uptime and team efficiency. New features like customizable reports and export options enable teams to conduct detailed reviews of metrics, such as incident load by time of day, incident classification, and response trends, supporting discussions on process improvements and enriching blameless retrospectives. By offering insights into missed incidents, incidents outside of SLA, and team member performance through an incident resolution leaderboard, PagerDuty's reporting updates empower operations teams to optimize their monitoring and response strategies effectively.

In the realm of DevOps, technical skills are often prioritized, but the importance of fostering a culture of empathy and effective communication is equally critical, especially during high-stress situations. The article discusses how soft skills, though harder to quantify, play a significant role in maintaining team harmony and operational success. It highlights communication frameworks like Nonviolent Communication and Crucial Conversations as effective tools in managing conflicts and enhancing understanding among team members. The piece emphasizes aligning on common values through systems like OKRs to facilitate cross-departmental collaboration and stresses the significance of choosing the right communication medium, particularly for distributed teams. Building inclusive relationships and considering diverse backgrounds can further strengthen team dynamics, and measuring these efforts through feedback and diversity metrics can provide insights into organizational health. The narrative underscores the role of empathy in creating a more supportive and effective work environment, drawing on examples from various sectors, including insights from FBI hostage negotiation tactics, to illustrate the transformative power of empathetic communication.

Opsmatic provides real-time infrastructure monitoring and has partnered with PagerDuty to enhance incident alerting and collaboration. The integration ensures that policy breaches or changes in system state, such as failed automation or unauthorized logins, automatically trigger incidents in PagerDuty. This collaboration allows for the synchronization of notifications, acknowledgements, and resolutions between the two platforms, facilitating quicker incident resolution and better visibility into infrastructure changes. Users can define tailored alerts and manually create incidents in PagerDuty from Opsmatic events, while documentation and tools are available to integrate Opsmatic webhooks into PagerDuty services.

Virtuoso, a luxury travel company, significantly improved its operations and reduced alert volume by integrating PagerDuty with the Event Enrichment Platform (EEP). Initially overwhelmed by thousands of daily alerts, the DevOps team implemented EEP to classify alerts as actionable or noise, enrich them with remediation information, and streamline them to PagerDuty for efficient response. This integration allowed the team to reduce their alert volume by 94% within three weeks and further decrease their daily alerts to just a few incidents. Weekly event triages helped suppress unnecessary noise, leading to enhanced situational awareness and faster problem remediation. As a result, Virtuoso's deployment and release processes accelerated, enabling the team to focus on automating their infrastructure and applying DevOps philosophies effectively.