May 2022 Summaries

In an ebook and video, learn how to quickly set up scalable infrastructure for startups transitioning into enterprises. Startup data teams can leverage the modern data stack to provide analytics value without increasing headcount or building infrastructure. This approach eliminates the need to build or maintain data pipelines and delivers analysis-ready data into a cloud data warehouse or data lake. A modern data stack will scale with your company's growth, reducing the need for constant engineering of your analytics stack.

The relationship between machine learning (ML) and database management systems (DBMS) is complex, with different strategies for integrating ML into DBMS depending on the composition of work. On one end of the spectrum are classic analytics teams using SQL data warehouses, where special SQL syntax for creating and evaluating ML models can be beneficial. BigQuery ML is an example of this approach. On the other end are data science teams primarily using machine learning libraries in Python, who may benefit from embedded DBMS like DuckDB. Many users exist somewhere in between these two extremes, with interoperability being a key challenge. Solutions such as Apache Arrow and Lakehouse aim to address this issue by allowing efficient exchange of relational data between systems. The popularity of each approach will depend on the specific needs and problems people are trying to solve.

Fivetran has launched High Volume Agent (HVA) connectors for Oracle and SQL Server databases, designed to handle high-volume replication with faster setup times. These HVA connectors minimize source load, allowing data teams to manage higher volumes and data structure changes more easily. The new connectors are beneficial in cases of databases that generate high data volumes or have a large repository of historical data. Fivetran's product principles emphasize ease of use from implementation to maintenance, with features such as schema mapping and compatibility with destination data types and formats.

A modern data stack, built on Fivetran and Google BigQuery, is essential for retailers to achieve the data visibility necessary for seamless personalized omnichannel experiences. Modern retailers face challenges such as increased adoption of omnichannel shopping, supply chain and inventory complexities, and customer expectations for personalization. A modern data stack allows fast access to trusted data from various sources, eliminating silos and increasing visibility across all operations. This results in benefits like a 360-degree view of the customer, enhanced personalization, faster insights, and more precise decision-making. Companies such as Chubbies and Nando's have successfully implemented modern data stacks to improve their business operations and marketing capabilities. A modern data stack also ensures high levels of security and privacy, meeting internal and regulatory requirements.

Forecasting time series data is essential for both individuals and businesses to anticipate future needs and manage resources effectively. Exponential smoothing is a popular time series forecasting method that assigns decreasing weights to past observations, emphasizing more recent data. There are three types of exponential smoothing: simple (for data with no seasonality or trend), double (for data with a trend but no seasonality), and triple (for data with both a trend and seasonality). The method is contrasted with simple moving averages, which weigh past observations equally. Simple exponential smoothing, which requires a single smoothing parameter known as alpha, is explained through its mathematical equation, highlighting the importance of selecting an appropriate alpha value for accurate predictions. The article provides a practical example of predicting car rental demand using simple exponential smoothing in Python, demonstrating its utility in real-world business scenarios.

Dustin Nascimento shares his experience working at Fivetran, a company that values employee development and rewards hard work. He highlights the importance of an entrepreneurial mindset, coaching for talent and leadership development, and finding a company that prioritizes growth and career opportunities. As a business development representative (BDR), Dustin emphasizes the need for effective training and support to succeed in the role and progress his career within the company. Fivetran encourages continuous learning and provides opportunities for employees to explore different departments, fostering an environment of growth and success.

Having a governance strategy provides businesses with data control and visibility, enabling them to discover new trends and gain insights from their data. This is achieved by having all data in one place, which allows for robust security measures and the ability to delegate access. Data governance, the combination of data control and data visibility, helps organizations democratize data usage within the company, improving overall data quality. Implementing a successful data governance strategy requires a framework that incorporates people, processes, technology, and an agile mindset. By focusing on positive business outcomes and encouraging healthy discourse among practitioners, businesses can achieve radical results in their data management efforts. Fivetran's plug-in data governance approach simplifies the implementation of data governance strategies within enterprises, promoting increased control and visibility across modern data stacks. Ultimately, data governance leads to a data-driven organization that can compete effectively in today's marketplace.

Fivetran has introduced webhooks, a feature that enables developers to use real-time messages for various purposes such as powering user experiences, transforming data, driving error alerting, and more. Webhooks work by automatically serving data without the need for repeated requests from users. This new feature eliminates uncertainty and extra effort in obtaining connector updates, allowing Fivetran to watch for specific events and push updates to a designated endpoint. Currently available events during beta include sync_start, sync_end, status, transformation_run_start, transformation_run_succeeded, transformation_run_failed, dbt_run_start, dbt_run_succeeded, and dbt_run_failed. Webhooks can be used to trigger various actions such as sending emails, Slack alerts, or in-product notifications when data is ready for consumption, or triggering notifications to DevOps tools like PagerDuty when a data pipeline breaks down.

A global research study reveals that 77% of enterprises lack real-time access to Enterprise Resource Planning (ERP) data, leading to poor business outcomes and lost revenue. The main challenges in obtaining real-time ERP data include system complexity, integration issues, incompatibilities, and a lack of in-house expertise. Software vendors further complicate the situation by intentionally making access to their solutions' data difficult. 85% of ERP professionals believe that stale data leads to incorrect decisions and poor business outcomes. Only 23% of companies have real-time ERP data access today, despite its numerous advantages such as faster decision-making, improved accuracy, increased efficiency, early problem detection, and smoother operations. Cloud-based ERP solutions and change data capture (CDC) can help enterprises harness the power of their ERP data in real-time. Log-based CDC is particularly effective due to its low impact on source systems and ability to enable real-time data movement. Implementing a real-time database replication solution with log-based CDC can provide business users timely access to ERP data without affecting the source applications.

Facebook Ads offers a powerful platform for promoting products to a vast audience, but effective advertising requires strategic planning and targeting the right audience with the right message. This guide outlines methods to sync customer data from a Snowflake data warehouse to Facebook Ads, focusing on creating targeted ad audiences based on email addresses. Three approaches are discussed: a quick manual method for one-off needs, an automated method using Facebook's Python API for regular updates, and a reliable method with Fivetran Activations for error-free automation. The manual method involves creating and uploading a CSV file of customer data to Facebook, while the automated approach leverages APIs to streamline data syncing. Fivetran Activations simplifies the process further by ensuring accurate and maintenance-free updates. Each method offers distinct advantages, from flexibility in manual updates to efficiency and reliability in automated processes, allowing advertisers to choose based on their specific needs and frequency of audience changes.

Change data capture (CDC) is a technique used to replicate data from databases and process it from various sources such as SaaS applications or systems accessible through APIs. It's gaining popularity due to its ability to drive powerful data-driven use cases, enable real-time analytics, data streaming, and machine learning. CDC allows for the transfer of data as changes happen, which is crucial for organizations operating critical systems 24/7 with no convenient time for batch windows. There are multiple types of change data capture methods, including log-based CDC, trigger-based CDC, CDC based on timestamps or another low watermark, and difference-based CDC. Fivetran is a modern data integration solution that leverages CDC to efficiently move high volumes of data between applications.

In the business realm, data is crucial, especially for marketing teams that depend on data teams to update information like contact properties and lead scores, but manual updates can be time-consuming and divert focus from more critical tasks. Bulk update functionality, particularly through tools like Fivetran Activations and HubSpot, can streamline this process by allowing efficient data refreshes and enabling data teams to focus on revenue-generating tasks. Fivetran Activations offers a robust solution for importing and syncing data from various sources into HubSpot, addressing challenges like dropped connections and data type conversions, and supporting scalable data synchronization. By leveraging reverse ETL, users can implement bulk updates in HubSpot through a simple eight-step process, involving tasks like connecting data warehouses, building models, creating syncs, and mapping fields, thus offering a more efficient and less labor-intensive alternative to manual bulk updates or complex ETL processes.

In a recent fireside chat, tech leaders discussed the modern data stack and its implications for businesses today. They highlighted SQL as the lingua franca of data analysis and emphasized the importance of Python in conjunction with SQL for effective data modeling. The conversation also touched upon the evolution of languages like SQL and Python, and how they have made programmers more productive and efficient. The emergence of the modern data stack was attributed to a change in cost, which unlocked ELT (Extract, Load, Transform) and enabled additional workloads. However, the panelists believe that the real challenge now is not reducing costs but improving usability. They also discussed the importance of governance solutions for managing complex systems and ensuring data security. The future of data apps was envisioned as autonomous applications that leverage predictive analytics to take actions based on data. The modern data stack is expected to be the foundation for these intelligent applications, which will become more business-critical in the coming years.

The high turnover among chief digital officers (CDOs) can be attributed to the challenging nature of data-driven digital transformation, which is more like a metamorphosis than a simple transformation. CDOs often face unrealistic expectations, lack of support from leadership, and resistance from employees due to skepticism about data analytics or inertia. To overcome these challenges, newly hired CDOs should start small and scale fast by targeting less ambitious but important business goals and promoting their successes internally. Employing the modern data stack (MDS) can help eliminate in-house building chores and lay a foundation for an efficient and reliable data and analytics strategy that improves future business outcomes and spurs innovation.

Agile development and DevOps have increased the need for effective data management, leading to the emergence of DataOps, a framework designed to handle the challenges of data-driven decision-making. Unlike DevOps, which focuses on software development, DataOps centers on building, testing, and releasing data products, requiring distinct skill sets and collaboration styles. Both frameworks share core principles such as automation, continuous improvement, and breaking down silos, but they differ in their focus: DevOps emphasizes fast, reliable software delivery, while DataOps aims to streamline data operations and enhance data quality. DevOps relies on well-established tools and practices to optimize software development, whereas DataOps, still evolving, seeks to improve data handling by adopting automation and learning from DevOps methodologies. Despite their differences, both frameworks foster collaboration, process simplification, and a customer-first mentality, ultimately aiming to deliver high-quality products and insights to end users.