February 2021 Summaries

In this blog post, Michael Kaminsky discusses a common pitfall in data science when using cloud data warehouses for machine learning tasks. The issue arises with untracked slowly changing dimensions, which can lead to poor real-world predictive accuracy of models. An example is provided where the goal is to predict customer churn in a subscription business. Two main pitfalls are identified: incorrect setup and the problem of slowly changing dimensions. To address these issues, Kaminsky suggests using a log format for data storage that tracks all changes made to different values in the database. Fivetran's "history mode" feature is mentioned as a tool to automatically record and capture changes to slowly changing dimensions, allowing for better model generalization from training to production.

The article discusses the importance of incrementally updating data pipelines instead of performing full syncs. Full syncs can take a long time and consume excessive network bandwidth, making them inefficient for routine updates. Incremental updates involve identifying changes made to a previous state of the source using methods like changelogs or last-modified timestamps. Changelogs contain a full history of updates, new records, and deleted records as a list of updates, while last-modified timestamps indicate when a record was last updated. However, challenges with incremental updates include scale, granularity of timestamps, late arrivals, and missing changelogs or timestamps. To overcome these issues, more performant code and efficient algorithms can be used to minimize sync times, using greater than or equal logic for timestamps, and identifying proxies for timestamps when necessary.

In 2020, Fivetran experienced rapid growth with doubled revenue and customer base, including leading companies like ASICS, Autodesk, and Square. The company expanded its workforce to nearly 500 and opened new offices in Germany, France, India, and Australia. Fivetran's product, always-available data pipelines, allows customers to extract data from any source and centralize it in a modern data warehouse. In 2020, the company added new connectors for enterprise databases and advanced its integration with Databricks. Fivetran was recognized as the top "EL" provider in the ELT architecture of the modern data stack and received several industry awards. The company is committed to creating a great place to work and expanding its team around the world while maintaining its core values: Integrity, Initiative, and 1 Team 1 Dream.

A cloud-based data warehouse can provide superior marketing analytics and a 360-degree view of customers. With the multitude of brand choices available, customer loyalty is challenging to achieve. However, understanding customers' buying motives allows businesses to anticipate their needs and influence purchase decisions. To do this, companies need to collect and analyze large amounts of data on their customers, which are typically stored in various line-of-business applications across the organization. By aggregating and analyzing such varied data, a customer 360 view is obtained, enabling businesses to better serve their customers and grow relationships with them, ultimately generating more revenue. In today's unpredictable business climate due to Covid-19, companies need to generate actionable insights that allow for quick changes in strategy. There are several ways to access the required data, each with its pros and cons. Centralizing data is key to better insights, which can be achieved through built-in dashboards in marketing technology (martech) tools or turnkey business intelligence (BI) dashboards like Domo. However, these options may not provide a complete view of customers or answer all questions about them. Building your own data warehouse from critical data sources within the company is an effective way to get the data needed for making business-critical decisions. While it requires more upfront investment than typical martech tools or turnkey dashboards, it allows businesses to leverage data from multiple sources and answer any question about their customers. To maximize the benefits of data in today's world, a data warehouse must handle the three V's of big data: volume, variety, and velocity. Cloud-based data warehouses like Snowflake, BigQuery, and Redshift meet these requirements and have an ecosystem of partners that integrate marketing data quickly and easily. The choice of data warehouse can significantly impact marketing efforts and company growth, making the technology selection crucial in today's fast-paced business environment.

In this week's Black History Month Spotlight, Fivetran introduces Samuel, a Source Partner Manager in their Oakland office. Samuel shares his enjoyment of the authentic culture at Fivetran and how it allows everyone to feel like themselves at work. If he didn't have a career in tech, Samuel would have considered being a firefighter due to his passion for helping people. He celebrates Black History Month by learning about the lives of influential Black figures from history and mentoring younger generations. One positive experience that has come out of the pandemic for him is spending more time with his parents, particularly caring for his father who became paralyzed while he was in college. Fivetran encourages those interested in working at their company to reach out as they continue to grow.

In 2021, business intelligence trends are expected to shift towards real-time data analysis and more flexible dashboards. The "death of the dashboard" refers to moving away from static presentation tools in favor of dynamic platforms that can answer questions on the spot. Data governance is also set to evolve into a balanced approach similar to Goldilocks' ideal situation, ensuring accessibility without compromising security. Real data will become more important as companies adopt modern data stacks for near real-time insights. The hype around AI may subside as businesses recognize the value of human interpretation and context in decision making. Cloud data exploration is expected to rise with a focus on self-service BI, while collaborative analytics will shift towards cross-team workshops for more effective problem solving. Finally, capturing elusive data sources like manager's notes or Excel files will become crucial for comprehensive analysis.

The blog post discusses the role of external schedulers and orchestration tools in optimizing data workflows, specifically focusing on Fivetran and dbt in the modern data stack. Fivetran automates data syncs between sources and destinations with customizable scheduling, while dbt handles scheduled data transformations once data is loaded into a warehouse. However, due to the sequential nature of ELT processes, timing issues can arise, which may hinder scalability and data quality. Orchestration tools like Airflow, which manage workflows through dynamic, extendable, and distributed systems, offer a solution by defining explicit task dependencies, thereby reducing latency and improving data quality. These tools allow for the seamless execution of tasks in parallel, ensuring dbt transformations only begin after Fivetran's data loading is complete. The post introduces an upcoming series focused on using Airflow for orchestrating Fivetran syncs and invites users to share their experiences with such orchestration in their data stacks.

Fivetran, a leading data integration platform, has announced its partnership with Census, an operational analytics tool. The collaboration aims to simplify end-to-end integration, improving operational efficiency and increasing revenue for customers. Fivetran is known for loading data from over 150 popular tools into warehouses, while Census writes data from common analytics warehouses and lakehouses into operational systems such as Intercom or Salesforce. The partnership will enable teams to change their operational systems on the fly without manual inputs, accelerating companies' ability to act on insights derived from their data.

In this week's Black History Month Spotlight, Fivetran introduces Marnae Cooper, a Senior Talent Operations Specialist who has made significant contributions in the data industry. Outside of tech, Marnae would likely pursue a career in art or media. She celebrates Black History Month by reading books from black authors and is currently enjoying Such a Fun Age by Kiley Reid. An unsung hero she wishes more people knew about is Ernie Chambers, a long-serving Nebraska state senator. A positive outcome of the pandemic for Marnae has been spending quality time with her family, including her mother who now works alongside her as a coworker. Fivetran is currently hiring and encourages interested candidates to apply.

In this week's Black History Month Spotlight, Fivetran introduces Joseph Long, a marketing campaign manager in their Oakland office. He enjoys working at Fivetran due to the company's core values and supportive team culture. If not in tech, he would have pursued a career in higher education. One of his favorite unsung heroes is Shirley Chisholm, who was the first black congresswoman and presidential candidate. A positive outcome from the pandemic for Joseph has been rediscovering his love for video games. Fivetran invites interested candidates to apply as they continue to grow their team across various regions and roles.

The article discusses the limitations of off-the-shelf Customer Data Platforms (CDPs) and suggests using data warehouses as an alternative solution. CDPs are databases that create a unified customer profile by pulling data from multiple sources, cleaning it, and making it available to other marketing systems. However, they have several shortcomings such as not being the single source of truth, not meshing well with data teams, lack of flexibility, owning your data, and not benefiting from the data ecosystem. The author argues that a data warehouse can be more effective as a CDP due to its ability to handle complex data models, provide unrestricted access to data, and benefit from the growing ecosystem around data warehouses. However, there are cases where an off-the-shelf CDP might make sense such as for vertical companies or those with no-code capabilities. The author also mentions tools like Hightouch that can help turn a data warehouse into a customer data platform.

The article discusses five key action points to achieve effective and reliable analytics dashboards. Firstly, it emphasizes eliminating build errors, delays, and bias in data presentation. Secondly, the importance of separating audience attributes from audience actions is highlighted, with customer behavior insights being more crucial than demographic information alone. Thirdly, periodic quality checks and audits are recommended to ensure data accuracy and reliability. Fourthly, compliance with changing rules and regulations related to data transfer, privacy, and storage is essential for maintaining legal standards. Lastly, sharing data across different players in an organization can create synergies and improve transparency and efficiency.

The article discusses the signs that indicate an organization needs to invest in a modern data stack. Key points include: 1) When the effort required to answer business questions starts to hinder productivity, it may be time to move to a modern data stack. This is often characterized by teams spending hours each month inputting data into spreadsheets and struggling with large file sizes. 2) Most growing teams will reach a point where they can feel the need for BI tools due to consistently hitting row limits and generating numerous CSV files. 3) The decision to invest in a data stack should be motivated by specific questions related to the customer journey that are currently time-consuming or impossible to answer without advanced data technology. These questions often require information from different systems across various business units, making manual analysis exhaustive for analysts and impossible for non-technical users.

In this week's Black History Month Spotlight, Fivetran introduces software engineer Rahel who shares her experience working at the company. She appreciates the supportive environment and learning opportunities provided by her colleagues. If she were not in tech, Rahel would explore new tech ideas with friends or consider studying accounting. To celebrate Black History Month, she recognizes successful black individuals and their journeys while also reflecting on her personal achievements. The pandemic has allowed her to improve her baking skills and manage her hair without a hairdresser. Fivetran is currently hiring for various roles across different teams and regions.

Databricks' release of Delta Lake has combined the best features of data lakes and data warehouses, creating what they call a "Lakehouse." This development brings together the cost advantages of data lakes with the efficient querying capabilities of traditional analytical databases. The ideal system should support multi-table transactions, table and column-level permissions, structured and semi-structured data, efficient joins, SQL and procedural programming languages, and advanced statistics. Regardless of whether this is called a cloud data warehouse, data lake, or relational database management system, the convergence between these two types of systems is significant for the future of data and analytics ecosystems.