May 2014 Summaries

Justin Liu from Crittercism discusses the importance of mobile app performance management, emphasizing the need for stability and responsiveness to meet user and business needs. As mobile apps become crucial for businesses, developers must ensure high uptime, defined as over 99% of app loads not crashing, and responsiveness, with service call latency under one second. Crittercism provides tools to monitor these metrics, utilizing error and service monitoring to identify and resolve issues, and integrates with PagerDuty for seamless alerts within existing DevOps systems. The complexity of mobile monitoring arises from diverse factors like operating systems, networks, and cloud services, which impact app performance differently across regions and devices.

PagerDuty is set to release updated versions of its iOS and Android apps, focusing on enhancing push notification functionality to address growing demand and customer feedback. The new updates will allow users to test push notifications and choose from 10 custom alert sounds to distinguish them from other app notifications. The company is addressing issues with device ID changes that can disrupt notification delivery by implementing code updates to capture and store the correct device ID more effectively. However, users are advised to maintain additional alert channels due to potential unreliability in push notification delivery times, as neither Google Cloud Messaging nor Apple Push Notification Service guarantees prompt delivery. The company encourages users to share further feature requests via email or Twitter.

Ville Saarinen, from Flowdock, discusses the challenges faced by software development teams when managing a multitude of tools for communication, version control, testing, deployment, monitoring, and customer support. The overabundance of services often leads to inefficiencies and information overload, as team members must frequently switch between platforms or manage excessive notifications. Flowdock addresses these issues with a team inbox that consolidates tool integrations into a real-time activity stream alongside the team's chat, fostering more organic discussions and reducing email clutter. This centralized approach enhances issue resolution by allowing rapid assignment and escalation of tasks, while also promoting knowledge sharing through transparent documentation and tagging of resolved problems, enabling teams to learn from past experiences. Saarinen emphasizes that open communication cultures, where information is freely accessible, empower teams to make informed decisions and distribute responsibility, although it requires careful management to prevent information overload.

Jason Skowronki, a product manager at Loggly, highlights the importance of effective alerting in the 24/7 digital economy, emphasizing how services like Loggly and PagerDuty work in tandem to help DevOps teams manage operational issues. Loggly, a leading cloud-based log management service with a broad customer base, enables system administrators and developers to troubleshoot and monitor system status, while PagerDuty provides timely alerts to address potential application problems that might affect revenue and profits. An illustrative scenario describes a sudden drop in signup rates, prompting an alert from PagerDuty, which leads to an investigation using Loggly's dashboard. This investigation uncovers a deployment issue affecting rendering in Internet Explorer, showcasing how Loggly's features, such as search, filtering, and dashboard visualization, allow teams to correlate alerts with system events, identify root causes, and resolve issues efficiently. The synergy between Loggly's detailed log analysis and PagerDuty's alert system ensures that DevOps teams can prioritize significant problems, minimizing unnecessary disruptions and speeding up problem resolution.

PagerDuty's implementation of Single Sign-On (SSO) received positive feedback from customers who appreciated the convenience of managing fewer passwords. The integration of OneLogin's Ruby SAML Toolkit was chosen for its reliability and compatibility with PagerDuty's Rails application. Extensive testing, including manual and automated checks, ensured the system's security and reliability, with specific modifications made to suit PagerDuty's needs. For mobile applications, SAML was combined with OAuth for authentication. The company conducted customer previews to refine their monitoring processes, primarily using Sumo Logic, and adjusted alerts to provide actionable and detailed information without compromising sensitive data. They addressed challenges such as clock drift, ensuring consistent authentication between servers. PagerDuty has partnered with identity providers like Okta and OneLogin to facilitate SSO setup, and offers guidance for implementing SSO with other SAML 2.0 capable providers, including Google Apps.

Chef is a versatile operations tool celebrated for its flexibility and automation capabilities, making it highly favored in organizations with extensive service deployments. During ChefConf, Ranjib Dey emphasized the importance of designing for failure, highlighting that crises are inevitable and advocating for making failures as "inexpensive and isolated" as possible. He discussed the significance of code design in mitigating failures, suggesting that leaner code is easier to test and replace. Ranjib highlighted unit testing as a fast method that captures a majority of errors, reinforcing codebase consistency and long-term maintainability. He underscored the need for communication to break down knowledge silos and foster collaboration between developers and operations, suggesting that good design thrives through shared knowledge. Ranjib also addressed the inherent complexity of modern codebases, likening them to a plate of spaghetti, and echoed the sentiment of embracing the complexity with a Zen attitude, as articulated by Netflix's Corey Bertram.

PagerDuty's team had a rewarding experience at Monitorama 2014 in Portland, which was highlighted by a strong sense of community, diversity, and openness in the tech industry. The event featured compelling talks, including Ashe Dryden's presentation on diversity and privilege within the tech world and James Mickens' humorous take on technology and its challenges. Attendees, mostly PagerDuty customers, provided valuable product feedback, allowing the team to gather insights on user experience improvements and pitch new features. The conference emphasized the importance of inclusive discussions alongside technical innovation, underscoring a collective effort to enhance monitoring practices and improve the quality of life for on-call professionals. The welcoming atmosphere and diverse representation at Monitorama contributed to a shared goal of improving system uptime and fostering a supportive professional community.

Erik Näslund, a back-end developer and operations specialist, shares his insights on the importance of effective monitoring in software development across companies of varying sizes. He highlights the challenges faced when monitoring is seen as costly or complex, often leading to oversight and customer-reported issues. Näslund describes experiences from different company sizes, noting that smaller teams often neglect monitoring due to resource constraints, while larger companies struggle with integrating flexible systems that are easy for developers to use. He emphasizes the necessity of both application performance management (APM) and real user monitoring (RUM) in ensuring website availability and diagnosing bottlenecks. Näslund introduces a hypothetical game, Awesome Space Battle, to illustrate the need for comprehensive monitoring, including event monitoring to ensure critical processes run smoothly. He advocates for integrating on-call schedules and notification systems to address issues promptly and stresses the importance of making monitoring tools accessible to developers, who are best suited to identify what needs to be monitored. Ultimately, Näslund argues that effective monitoring should be a fundamental part of the development process, akin to unit testing, to preemptively address potential issues and maintain application reliability.

PagerDuty introduced a referral program called ReferralDuty, which incentivizes users to refer new customers by offering a chance to win a free pass to the AWS re:Invent conference in Las Vegas from November 11-14, 2014. To participate, individuals need to provide the contact information of a friend who is not a current PagerDuty customer, not in an active sales negotiation with PagerDuty, and has not been referred by another source. The referred person must enter into a free trial and be accepted by PagerDuty, with the final decision resting solely with the company.

PagerDuty engineers prioritize reliability in their mobile apps, focusing on maximizing resiliency and minimizing downtime for users. After redesigning their mobile applications, they encountered challenges with their build and release process, which initially used Apache Cordova to create hybrid apps with a Javascript core. This approach allowed code reuse across platforms but introduced complexities, particularly when integrating modern build tools and managing dependencies. By borrowing concepts from software artifacts used in other programming languages, they improved their build process, reducing errors and enhancing efficiency. The team also explored using npm for dependency management and Github's Release API to handle build artifacts, allowing for a more streamlined and flexible release process. Despite the lack of a standardized method for packaging front-end Javascript applications, the engineers embraced the adaptability of Javascript, leveraging the simplicity of a single HTML file that loads necessary resources and is compatible with various platforms. This approach allows for quicker, more reliable releases while maintaining flexibility in their development workflow.

Trevor Parsons, Chief Scientist and Co-Founder at Logentries, emphasizes the importance of real-time log data analysis to monitor business activities and user events effectively across an organization. Traditional logging solutions, which operate on a delayed schedule, are insufficient for immediate alerting in critical situations. He recommends setting specific alert thresholds to discern the urgency of events and customize notifications for different types of occurrences, such as exceptions, errors, response times, resource usage, and significant business events like customer sign-ups or new feature adoption. By leveraging tools like Logentries, organizations can integrate with platforms like PagerDuty to receive timely alerts and maintain visibility over both problematic and positive developments, enhancing team awareness and morale.