September 2019 Summaries

We built our own CI infrastructure due to growing needs beyond CircleCI's capabilities, specifically the need for physical device testing and long-running functional stress tests. We used Anka virtualization technology on top of Hypervisor.framework to create isolated environments, and Buildkite as a modern frontend for scheduling and managing builds. Packer automated the creation of virtual machine images, which are stored in Anka's Registry and cloned by build agents. The setup allows for easy scalability, reduced flakiness, and flexibility in running long-running pipelines on physical devices or remote machines.

Faced with the limitations of CircleCI, such as the inability to run UI tests on physical devices and the costliness of running lengthy internal testing tools, a team decided to establish its own continuous integration (CI) infrastructure. They opted for Veertu's Anka, a virtualization technology for macOS that allows builds to run in isolated environments, akin to Docker containers, despite Docker's lack of support for macOS containers. By using tools like HashiCorp's Packer to automate virtual machine creation and Buildkite for managing build pipelines, the team constructed a scalable and flexible CI system that can run multiple pipeline steps simultaneously across various machines. This setup allows for greater control, reduced flakiness, and the capacity to incorporate physical devices into testing, addressing the team's evolving needs as it grows.

The text discusses various mobile test automation tools, each with their strengths and weaknesses. TestComplete is highlighted for its object recognition engine and AI-powered visual recognition, while Tricentis Tosca supports a wide range of testing types and has a comprehensive platform. Ranorex offers an all-in-one environment for test automation, supporting multiple platforms, languages, and technologies. Katalon Studio provides a free, codeless solution with powerful features like Smart Xpath Generator and Smart Wait. Appium Studio is a simple yet powerful tool developed by Experitest, while TestProject has a community-driven repository of readymade coded actions or "add-ons". Instabug focuses on user-centric stability and performance monitoring for mobile apps, offering a sandbox and demo options to explore its capabilities.

Mobile test automation tools are essential for improving testing efficiency, increasing code coverage, and enhancing testers' productivity as app complexity grows. Tools like TestComplete, Tricentis Tosca, Ranorex, Katalon Studio, Appium Studio, and TestProject offer diverse features such as record and replay test creation, AI-powered recognition, and support for multiple testing types and platforms. TestComplete, for instance, supports various scripting languages and integrates with numerous CI tools, while Katalon Studio provides a free, codeless automation solution with AI-backed features. Although these tools generally require Windows, they offer capabilities like integration with CI/CD tools, detailed analytics, and REST APIs for custom integrations. Pricing varies across tools, with some offering free versions with limited features and others requiring contact for quotes or tiered pricing based on functionality and support.

Google's HEART framework is a user experience (UX) metrics approach that was developed by human-computer interaction researchers to bridge the gap between user-centered and data-driven metrics. The framework measures signals for Happiness, Engagement, Adoption and Retention, and Task Success, which are grounded in simplicity and knowledge of how to elegantly interpret quantitative metrics. To get started with the HEART framework, one must identify goals, choose relevant signals, and select specific, measurable metrics that can be tracked over time on a dashboard. The Google HEART framework is flexible enough to accommodate new methodologies as they develop and provides guidance through the Goals-Signals-Metrics process. By applying the HEART framework, mobile app owners can confidently track their progress with a focus on the right metrics for the right reasons.

In the early 2000s, Google's researchers developed the HEART framework to enhance user experience measurement by integrating data-driven and user-centered metrics, which was a shift from business-centered goals. This framework focuses on Happiness, Engagement, Activation, Retention, and Task success, allowing app developers to track user experience quantitatively and qualitatively. The process involves setting clear goals, identifying specific signals to gauge success, and selecting measurable metrics relevant to the app's objectives while considering user behavior and attitudes. Google's Goals-Signals-Metrics approach guides app developers in choosing metrics that align with their goals, ensuring that insights are actionable. The framework's adaptability and comprehensive nature make it a valuable tool for evaluating user experience, as demonstrated through various Google case studies, such as updates to Google Maps and Gmail, where user feedback and behavior informed design decisions and improvements.