June 2021 Summaries

Automatic data capture, when implemented correctly, can enhance data governance by making it an integral part of the data architecture rather than an afterthought. This approach, as advocated by Heap, counters the misconception that manual tracking ensures better governance and avoids data chaos. Automatic capture simplifies the process of managing large datasets by collecting all interactions and behaviors from the outset, which manual methods struggle to achieve due to their reliance on spreadsheets and limited event tracking. Despite fears of generating unverified data, autocapture systems can effectively organize and verify data from the moment of definition, providing a comprehensive and reliable dataset. This allows teams to make data-driven decisions and discover insights that may have been missed with manual tracking, thus supporting the development of innovative digital products and experiences.

For those planning a relaxing summer, whether on the beach or a quiet staycation, diving into digital analytics literature could be as engaging as a captivating novel. Digital analytics provides insights into user behavior on websites or products, offering a data-driven approach to decision-making that surpasses gut instincts. Recommended reads include "The Lean Product Playbook" for understanding product-market fit, "Naked Statistics" for a refreshing take on statistical tools, and "The Data Warehouse Toolkit" for foundational knowledge in data engineering. Books like "Keeping Up With the Quants" and "Lean Analytics" emphasize quantitative literacy and using analytics for startup success, while "The Art of Thinking Clearly" and "How to Increase Revenue by Measuring Customer Behavior" focus on overcoming biases and leveraging user interaction data for revenue growth. Additionally, the StitchFix Algorithms Blog and The Digital Analytics Power Hour podcast offer practical insights into applying data science in business. For those short on time, following industry experts on social media can also provide valuable insights.

Heap utilizes Apache Spark to enhance its Heap Connect product, which facilitates rapid data transfer to customer warehouses, but encountered performance issues due to a slowness in Spark's ORC file reader. The problem arose from an exponential algorithmic issue in processing binary trees, leading to significant delays in query planning within Spark's Driver component, causing Executor idling. The team addressed this by restructuring their tree representation to avoid deep, skewed trees, resulting in substantial performance improvements. They circumvented the need for immediate changes to Spark's internals, thereby accelerating their fix deployment, and ultimately contributed a solution to Spark, eliminating the exponential complexity in ORC predicate conversion, which was incorporated into the Spark 3.0 release.

Digital experiences are rapidly transforming interactions, particularly in the business world, where they often serve as the primary or sole touchpoint between companies and customers. Despite the importance of creating compelling and user-friendly digital experiences, there remains a significant disconnect between what digital teams perceive and what users actually experience, with many teams lacking objective insights into user behavior. Traditional analytics tools have failed to bridge this gap, as they often provide data without meaningful insights. Heap Illuminate proposes a paradigm shift by using data science to offer proactive analytics tools, such as Effort Analysis, Step Suggestions, Group Suggestions, and Pageview Suggestions, which automatically highlight critical user behavior patterns and friction points. This approach aims to provide teams with comprehensive, actionable insights that enhance the digital experience by ensuring all relevant user data is captured and analyzed effectively. Heap's platform promises to deliver a complete and reliable digital data foundation, combined with advanced data science capabilities, to help businesses optimize customer journeys and adapt to the evolving digital landscape.

Heap has partnered with Flagsmith to enhance the efficiency of A/B testing by integrating feature flag data with customer behavioral insights, allowing teams to optimize digital experiences based on user needs. This collaboration enables product teams to decouple deployment and release processes, allowing code to be pushed to production multiple times a day while selectively releasing new features to specific user segments for testing. The integration allows for real-time tracking and analysis of feature performance without requiring engineering intervention, facilitating rapid iteration and informed decision-making. Flagsmith's feature flags can be managed across different platforms, and when combined with Heap's data analysis tools, teams can run scientifically-driven A/B tests to fine-tune user experiences. This joint solution enhances the ability of product teams to deliver optimal user experiences by utilizing comprehensive behavioral data and feature flag management, benefiting over 6,000 businesses already leveraging Heap for improved product development and user engagement.