December 2020 Summaries

A modern data stack (MDS) can significantly improve an organization's analytics capabilities by reducing engineering costs, increasing data literacy, and simplifying data engineering tasks. Key components of an MDS include automated data integration, a cloud data warehouse for rapid query speeds, and a business intelligence tool for easy data visualization and report generation. Real-world examples from Fivetran customers demonstrate the benefits of an MDS across seven important metrics: reduced data engineering overhead, increased data team capacity, ability to execute new data projects, decreased reporting time, improved infrastructure reliability, enhanced data literacy, and increased strategic performance metrics.

Clari, a revenue operations platform, uses a modern data stack to provide valuable insights back to its customers. The company's one-man analytics department, led by Gary Sahota, automates data integration and saves time by using Fivetran, Snowflake, and Tableau. This approach enables the team to drive product growth with data and encourages employees to use insights to influence the product itself. By focusing on analytics as a core function and strategic investment, Clari aims to generate revenue through data-driven decisions and new product features.

To improve data analytics, businesses should focus on three essential steps: agreeing on an internal system of record, consolidating data in a cloud data warehouse, and creating a solid dimensional schema. Establishing a single "system of record" helps resolve discrepancies and prevent complexity by designating one location as the authoritative source for each data concept. Consolidating data in a SQL data warehouse allows for efficient querying and management of structured data, facilitating decision-making processes. For larger companies, building a dimensional schema simplifies the data model, reducing complexity and creating a unified view of data to support various analytical needs. This approach enables data-driven decision-making without requiring sophisticated high-tech engineering, emphasizing the need for a robust data infrastructure and a leadership team willing to adapt based on emerging data insights.

Emery Sapp and Sons, a heavy civil construction firm, has successfully implemented a modern data stack to improve its analytics capabilities and support business growth. The company's previous on-premises SQL server setup was not flexible enough to scale with the fast-growing organization. By adopting Google BigQuery, dbt, Fivetran, and Looker, Emery Sapp and Sons has automated data integration, reduced maintenance requirements, and empowered other departments with self-service options and high data quality. The new data stack allows the company to handle its growing data needs more efficiently and paves the way for future advancements in machine learning and predictive analytics.

Kate Srachnyi, founder of Story by Data, shared her top recommendations for enhancing data visualizations during the Modern Data Stack 2020 Conference. She emphasized designing with intentionality, reducing clutter, and using color intentionally to create informative, appealing, and efficient visualizations. Designing with the audience in mind, removing unnecessary elements, and utilizing colors effectively are crucial for telling a clear data story. Srachnyi also highlighted the importance of testing visualizations with new users and adjusting accordingly.

Direct-to-consumer and e-commerce companies face significant challenges in managing their data due to the proliferation of disparate systems, similar to the complex data infrastructure of industry giants like McDonald's. With mission-critical data scattered across numerous platforms, these companies struggle to create a unified customer view, often resulting in conflicting or incomplete data interpretations. Solutions like Fivetran, Snowflake, and Latticework Insights offer modern data stack advancements that simplify data retrieval, storage, and analysis, enabling businesses to derive actionable insights. The integration of these technologies is essential for addressing issues such as attribution, lifetime value versus acquisition cost analysis, and consolidated data visualization, ultimately allowing companies to transition from manual, fragmented approaches to streamlined, data-driven decision-making.

Building an analytics team at a high-growth company requires focus, understanding priorities, and establishing trust. Jacob Bedard, Director of Data Analytics at Dialpad, emphasizes the importance of laying the right foundation for analysis and ensuring stakeholders feel confident in basic reporting. A modern data stack consists of automated data pipeline tools, cloud data destinations, post-load transformation tools, and business intelligence engines. Key lessons include earning and keeping trust, prioritizing "nightmare scenarios," being nice to your future self by considering scalability, and embracing change.

Fivetran has announced the availability of its services in Canada and the United Kingdom, aiming to help businesses comply with data localisation requirements wherever they operate. The company recognizes that data is crucial for business operations and aims to provide quick and reliable access while ensuring security concerns are addressed. By allowing customers to set their preferred region, Fivetran enables British and Canadian clients to meet both internal and external security and local regulation requirements while enjoying automated data integration benefits.

Huel, a direct-to-consumer company known for its nutritious meals and fun branding, uses data to understand customers and improve their experience. Data Scientist Jay Kotecha and Ecommerce Director Ollie Scheers work closely together to ensure the company follows its core value of "Make Customers Happy." They analyze customer data from various sources to create a holistic understanding of the Huel customer journey and use that knowledge to enhance the business. The company's modern data stack allows them to analyze customer experience, sales, marketing, and subscription data, helping identify high-value customers and addressing pain points in near-real-time. Their future aspirations include using machine learning for predictive modeling and natural language processing to gain rapid insights from customer feedback.

Fivetran's History Mode is a feature that enables historical analysis by tracking changes made to table columns. It achieves Type 2 Slowly Changing Dimensions (Type 2 SCD) by adding a new timestamped row for every change. Unlike other methods, Fivetran automates the process of building and maintaining historical analysis, allowing businesses to focus on unique challenges. While Salesforce's native field history tracking has limitations such as a 20-field per object limit and separate storage of historical data, Fivetran keeps all historical data in one table with no limit. Designing SQL transformations using tools like dbt can also be used for building table history but requires more complex setup and queries. Fivetran's History Mode supports tracking deleted data out-of-the-box and is easier to use, making it a valuable tool for historical analysis.

The Modern Data Stack 2020 Conference featured a keynote panel discussion on the future of data tools and technology. Fivetran CEO George Fraser led the conversation with four industry experts, including Michell Ufford, Martin Casado, Bob Muglia, and Tristan Handy. Topics covered included the role of data lakes in modern data stacks, how machine learning will influence analytics, and potential use cases for the modern data stack. The panelists shared their insights on these subjects, with some predicting a shift towards decentralization of data teams or a "data mesh," while others believed that SQL data warehouses would replace data lakes in the future. They also discussed the possibility of an open-source file format for sharing between different use cases and the potential for machine learning to be integrated into analytics tools. The panel concluded by exploring various use cases, such as complex medical data and autonomous decision-making applications, that may become more prevalent within the modern data stack in the coming years.

Apache Kafka is an open-source message broker that has gained popularity due to its ability to scale to large numbers of messages. It is often used to decouple producers and consumers of data, such as in the case of Fivetran where it buffers customer-generated webhooks before loading them into a data warehouse. However, some proponents argue that Kafka can replace traditional relational databases as the definitive record of events. This architecture, known as "turning the database inside out," involves appending events to Kafka and reading from downstream views representing the present state. While it is possible in theory to implement this architecture with both reads and writes, doing so would require addressing every hard problem faced by traditional database management systems (DBMS). This includes dealing with issues like dirty reads, phantom reads, write skew, and other symptoms of a hastily implemented DBMS. The fundamental issue with using Kafka as the primary data store is that it provides no isolation. Isolation means that all transactions occur along some consistent history. Without proper isolation, anomalies can arise in scenarios like inventory management for an online store, where two users could end up purchasing the same item due to outdated inventory readings. To avoid these problems, Kafka can be used alongside a traditional database. OLTP databases are good at admission control of events and ensuring only consistent streams of events are emitted. They can also handle millions of transactions per second. Using change-data-capture (CDC) with an OLTP database allows for event generation while maintaining an operational story in recovery scenarios. In conclusion, real-time streaming message brokers like Kafka are useful tools for managing high-velocity data, but traditional DBMSs are still needed for proper isolation and transaction management. The best approach is to use OLTP databases for admission control, CDC for event generation, and model downstream copies of the data as materialized views.

ETL (Extract, Transform, Load) tools are designed to simplify the process of data integration by abstracting away many considerations and providing an easy-to-use interface. Modern cloud-based ETL tools like Fivetran offer features such as simple setup, automated updates, schema drift handling, infrastructure management, and robust security measures. These tools can help businesses save time and resources while ensuring reliable data access and accelerating data analysis.

Fivetran Inc. has introduced History Mode, a feature that enables data analysts to analyze historical data in just a few clicks by turning on the feature for specific tables. The Type 2 Slowly Changing Dimensions (Type 2 SCD) approach adds a new timestamped row for every change made to a column. Fivetran customers have found various use cases for History Mode, including retaining history of systems of record, tracking account growth over time, understanding the history of bookings and cancellations, analyzing the impact of subscription changes, and tracking customer success, alliances, and other employee influence on revenue. The company offers resources to learn more about History Mode and a free 14-day trial for users to sync their Salesforce data and enable the feature.