April 2017 Summaries

The Internet of Things (IoT) is increasingly becoming integral to both personal and enterprise environments, necessitating robust incident management solutions to handle the challenges posed by the vast array of IoT devices and the data they generate. Effective IoT systems require comprehensive monitoring and integration, which can be facilitated by tools like PagerDuty, enabling centralized management and customizable workflows through its API. The platform's capabilities are enhanced by allowing custom integrations, such as embedding dashboards and transforming data into actionable incidents, which are crucial for managing the complexities of IoT environments. Various applications, like Resin.io and Temboo, demonstrate the versatility and innovation that PagerDuty's APIs bring to IoT incident management, while tools like SmartBear’s AlertSite complement its functions with automated testing and synthetic monitoring. Given the potential for IoT devices to be exploited for malicious purposes, incident management becomes vital for safeguarding users and ensuring the reliability and safety of IoT applications, highlighting the importance of preparing for the IoT-centric future with platforms like PagerDuty.

In today's interconnected digital economy, organizations must navigate the benefits and risks of integrated IT infrastructures, which enhance innovation but also increase vulnerability to cyber threats. To mitigate these risks, companies need a comprehensive incident response plan that involves all key departments, including IT Infrastructure, Development, Quality Assurance, Public Relations, Legal, Human Resources, Customer Service, and Risk Management. This plan should be preemptively developed to facilitate a swift and coordinated response to incidents, ensuring that leadership support aligns with company-wide efforts. Public Relations manages external communication, Legal addresses liability and compliance, HR ensures the right personnel and policies are in place, Customer Service maintains communication with users, and Risk Management collaborates with IT to identify and mitigate potential threats. An effective incident response is not just the IT department's responsibility but requires a concerted effort across all organizational functions to proactively manage and prevent incidents.

The text discusses strategies for professional development and career planning, emphasizing the importance of setting specific, attainable goals for the future using the SMART method, which ensures goals are specific, measurable, achievable, results-focused, and time-based. It suggests breaking milestones into manageable tasks, investing in personal growth, and sharing career plans with mentors or peers to enhance accountability and success. The text stresses the value of self-reflection on a regular basis to assess progress, appreciate achievements, and identify actions for improvement. Consistency in applying these strategies is highlighted as key to long-term success, along with remaining open to change and seeking enjoyment in the process.

Code reviews are a crucial component of modern software development, but they can often be counterproductive and demoralizing due to a lack of clear guidelines and effective communication. This text emphasizes the importance of understanding the purpose of code reviews, which can range from bug detection to knowledge sharing, and encourages teams to have explicit discussions to align expectations. It advocates for automating style checks to focus reviews on substantive issues rather than formatting, and suggests offering suggestions rather than directives to foster constructive dialogue. The use of positive language can mitigate the negative bias in written communication, making reviews less contentious and more collaborative. Furthermore, code reviews should be viewed as learning opportunities for both the reviewer and the author, promoting a deeper understanding of the codebase. When receiving reviews, it's important to interpret terse comments with generosity, recognizing that they may be poorly phrased suggestions rather than criticisms. Overall, the text suggests that a kinder, more collaborative approach to code reviews can enhance productivity and morale.

PagerDuty has open-sourced a tool called DNSmetrics, which they developed to gather and transform metrics from their managed DNS providers for monitoring and alerting purposes. DNS is critical for translating domain names into network addresses, ensuring service availability, and mitigating issues like DoS attacks. To enhance resilience, PagerDuty utilizes two DNS providers in an active-active configuration, but found monitoring these providers challenging due to differing metrics and interfaces. DNSmetrics addresses this by connecting to both providers via APIs, normalizing the metrics, and emitting them in a standard format for integration into time-series monitoring systems. This consolidation allows for a comprehensive dashboard and flexible alerting across all DNS providers and zones. The tool, initially supporting only PagerDuty's providers, is designed for easy extension to others with REST APIs. DNSmetrics is available for use via a Docker container, inviting others to integrate it into their environments.

Reducing alert noise is essential for improving team efficiency and satisfaction, and this process can be streamlined using the right tools and workflow. A holistic view of your alerting infrastructure is crucial for identifying non-actionable, noisy alerts, which can then be managed without altering upstream tool configurations. PagerDuty's Infrastructure Health Application and Alerts Table provide a comprehensive visualization and detailed information about alert sources and severity, enabling users to pinpoint and suppress unnecessary alerts effectively. By using PagerDuty’s Event Rules Engine, teams can suppress non-critical alerts without changing upstream configurations, maintaining access to all data for a complete view of their infrastructure. Achieving the right balance in alert suppression involves both art and science, with comprehensive information aiding in focusing on critical issues. PagerDuty's solutions empower development and operations teams to manage production services efficiently, with new capabilities designed specifically for developers enhancing this process.

PagerDuty has introduced the On-Call Timeline feature across its web interface, iOS, Android, and Apple Watch apps to address the challenges users face in managing complex on-call schedules. This tool provides a centralized view for users to easily track upcoming on-call shifts, visualize when they go on or off call, understand the escalation policies and levels for which they are responsible, and see the remaining time in their current or upcoming shifts. It also allows users to export their on-call schedules to compatible calendars like Google Calendar and Outlook. Offered at no additional cost with all PagerDuty plans, the On-Call Timeline aims to streamline on-call management across all devices, with full details available in the company's support documentation.

PagerDuty has partnered with Atlassian to enhance incident management solutions for developers by integrating its services with Atlassian's HipChat, a tool designed to boost team productivity through improved communication. The new PagerDuty HipChat Extension enables developers to receive rich incident notifications directly in HipChat rooms, perform actions such as acknowledging and resolving incidents via the sidebar, and utilize ChatOps commands to manage incidents without leaving the chat platform. This integration aims to streamline incident response processes by reducing the need to switch between apps and ensuring secure access through simple sign-ins. It also allows for flexible setup, enabling the mapping of multiple business services to specific HipChat rooms to ensure appropriate alerts reach the correct response teams. Early adopters, like Expedia, have praised the extension for replicating core functionalities that were previously managed internally, highlighting its ability to provide context and control directly within chat environments.

Chatbots play a crucial role in leveraging ChatOps to optimize monitoring and incident management workflows, particularly within organizations adopting DevOps practices. These programs integrate with communication platforms like Slack or HipChat, allowing for automated monitoring and incident response through human input, thus enhancing operational efficiency. By allowing actions to be executed directly from chat interfaces, chatbots eliminate the need to switch between tools, thereby maximizing visibility and facilitating real-time operations. This integration supports seamless communication and operational agility, which are core principles of DevOps, ensuring teams can act continuously and address issues promptly before they disrupt the continuous delivery pipeline. Chatbots thereby enhance instant, visible communication and efficient responsibility dissemination among team members, making them invaluable for modern IT and development operations.

David Yang, a senior engineer at PagerDuty, discusses the transition from a monolithic codebase to a microservices architecture where developers own their services, highlighting both the benefits and challenges of this shift. The new model fosters greater developer independence, reduces cycle times for addressing issues, and encourages a culture focused on eliminating roadblocks. This approach has clarified service ownership, making it easier to manage operability issues and streamline the incident response process. However, it also introduces challenges such as the need for ongoing operational maintenance and the potential for knowledge gaps, particularly with legacy systems. To address these issues, PagerDuty is focusing on leveraging KPIs and metrics to make objective decisions and balance operational work with feature development, ultimately aiming to deliver consistent customer value through collaboration and effective prioritization.

The role of software developers has evolved significantly, expanding beyond writing code to include ideation, testing, securing, deploying, maintaining, and supporting software, as emphasized by PagerDuty's experience. PagerDuty, undergoing a transition over two years, aimed to integrate development and operations by distributing operational ownership across all developers, thereby fostering a more agile and nimble environment. This shift led to substantial improvements, including a 45% increase in changes deployed to production and a 50% reduction in Mean Time To Resolution of major incidents from 2015 to 2016. Empowering developers with ownership and accountability for their code, supported by automation and streamlined processes, enhances their ability to improve product quality and customer satisfaction. As technology teams face demands for faster delivery cycles, developers are increasingly positioned as owners of the end-user experience, which simplifies ownership and boosts efficiency.

In the evolving role of software developers, being on-call to manage production services has become an integral part of their responsibilities, fostering skills in building resilient systems and enhancing customer empathy. PagerDuty has developed new tools and APIs to streamline the on-call process, reducing anxiety and improving efficiency for developers. The On-Call Timeline visually represents shifts and escalation policies, while the Events API v2 integrates and normalizes event data from various monitoring systems. The Operations Command Console and Infrastructure Health Application aid in identifying and responding to issues, and the enhanced Incident Creation and Merge APIs allow for efficient incident management. Additionally, PagerDuty offers manual incident creation through its mobile app, ensuring developers can address issues promptly from anywhere. These features reflect PagerDuty's commitment to improving the on-call experience for developers, drawing from its own insights as a company familiar with the demands of being on-call.