October 2021 Summaries

Heap faced performance challenges with its multi-petabyte Postgres database cluster due to rapid data growth during COVID, which led to storage issues and performance degradation. The company identified that the utilization of SSDs near capacity, especially with the ZFS filesystem's copy-on-write (CoW) operations, was causing the slowdown. To address this, Heap explored upgrading to ZFS 2.x with Zstandard compression, which promised a better compression ratio and significant disk space savings compared to the existing lz4 compression. Rigorous testing revealed that the switch to Zstandard maintained query performance while reducing storage usage by 20% and halving average write durations. Consequently, Heap rolled out the new configuration across its database nodes, resulting in substantial storage savings and improved write performance, demonstrating the impact of minor configuration adjustments at scale and the value of leveraging open-source software solutions.

Building a company involves the crucial task of assembling a team, which becomes particularly challenging when hiring for roles outside one's expertise. The author, with a background in software engineering, describes a system used at Heap to tackle this issue, emphasizing the importance of conducting thorough research to understand the role, identifying potential pitfalls, and constructing an interview process that emulates the job. This approach includes leveraging external expertise for areas beyond the team's knowledge and ensuring that the candidate can effectively communicate their approach and trade-offs during the interview. The post also highlights the significance of hiring individuals who can build out a new function from scratch, emphasizing the need for candidates who have seen high-functioning teams before and are eager for leadership opportunities. Additionally, creating a 30-60-90 day plan can help align expectations and ensure that the candidate feels valued and understands their strategic importance. For some central roles, it may be beneficial for the founding team to initially perform the functions to gather valuable market insights before hiring.

Heap, a B2B SaaS company, tackled design debt and usability challenges through a focused two-week sprint called "Extreme Makeover: Heap Edition," aimed at improving the product experience for new users. The initiative involved pausing other ongoing projects to prioritize quick wins, which were defined as changes that could be implemented in under two engineering days and would enhance usability. This company-wide effort fostered camaraderie and included fun elements such as the Emoji Olympics to boost engagement and morale. The sprint resulted in over 40 feature launches, bug fixes, and product improvements, significantly enhancing Heap's usability and empowering teams to continue addressing usability issues in the future. Feedback from users and internal teams indicated a notable improvement in the product experience, encouraging ongoing efforts to manage design debt. This structured approach to quick wins is set to be a cornerstone of Heap's strategy for continuous product enhancement.

Audit tables, often used to track changes in database tables, proved invaluable in maintaining the stability of a distributed Postgres cluster operated by Heap, a product analytics tool. When a schema migration in the cluster led to ingestion latency issues due to a bug in Node code treating a string as a number, audit tables enabled a swift resolution by allowing the quick restoration of deleted metadata entries. This incident highlighted the utility of audit tables not only in logging changes but also in facilitating the reversal of problematic changes, thereby preventing severe disruptions. The creation of audit tables involves setting up ordinary tables with additional columns to track changes, supported by triggers to capture data modifications. While they introduce some overhead, audit tables are particularly useful for metadata tables containing crucial information, making them a valuable tool for systems requiring reliable data management and quick troubleshooting capabilities.

Funnels have traditionally been used to map out customer journeys, but their linear nature often fails to capture the true complexity of user behavior, leading to misguided insights and wasted resources. Recognizing these limitations, Heap introduces Journey Maps, a new feature within Heap Illuminate that offers a more nuanced understanding of user paths by tracking multiple routes to the same goal and assessing the impact of optional steps. This tool helps businesses identify which user behaviors lead to higher conversion rates by comparing the outcomes of different user choices, offering concrete evidence on how to optimize digital experiences. By revealing the real paths users take, Journey Maps empowers teams to make informed decisions that enhance customer engagement and improve business results, marking a significant advancement over traditional funnel analysis methods.