July 2023 Summaries

We're excited to announce that Fivetran now offers a new data lake destination in Beta: Delta Lake on Azure Data Lake Storage (ADLS). This new destination allows customers to unify all their data in one place, the "lakehouse", with minimal effort and cost. Fivetran automates the process of extracting, cleansing, conforming, and converting data to Delta Lake format, before loading it into ADLS, making advanced analytics and AI on the data faster. The new destination is a big deal because it simplifies the process of landing data from multiple sources into ADLS, reduces headaches with change data capture, and provides native integration with Databricks Unity Catalog and Microsoft Purview for governance and lineage tracking. With Fivetran's platform being completely no-code, changes to source data are reliably synced, and a single connector can be set up in under 10 minutes. The new destination also features less expensive ingestion using Fivetran compute, instead of Databricks compute, making it more cost-effective for customers. Ultimately, Fivetran aims to support data ingestion from all sources onto any cloud, in any analytics format.

The European Commission has announced an adequacy decision for the United States regarding the EU-U.S. Data Privacy Framework, making data movement between the two regions more straightforward and compliant with GDPR standards. This decision allows global enterprises to move and use their data internationally more easily. The EU-U.S. Data Privacy Framework also enables legal redress for EU individuals in cases of suspected data mishandling by US-based entities through a Data Protection Review Court. Companies transitioning from the old EU-US Privacy Shield standard can automatically move to the new framework, provided their privacy policies are updated and they meet DPF standards. Fivetran services are GDPR-compliant and will be transitioning to the new Data Privacy Framework.

A well-engineered data pipeline can significantly impact a company's efficiency and cost savings. Key factors to consider include data normalization, which ensures accurate and consistent data; idempotence, which prevents duplicate data creation during sync failures; incremental syncing, which reduces data volume and network bandwidth consumption; granular data selection, allowing teams to de-select unnecessary tables and columns; and programmatic pipeline management, enabling efficient automation of workflows. By selecting a data pipeline that incorporates these features, companies can save time, control infrastructure costs, and improve overall efficiency.

The modern cloud has enabled significant advancements in data movement and governance, allowing organizations to expand their production possibilities frontier and reduce scarcity. The shift from ETL to ELT has eliminated the need for bespoke solutions and labor-intensive transformations, instead enabling flexible and automated pipelines that save both money and engineering talent. This has also solved a long-standing tension in data science, where data scientists were often forced to spend time on upstream technical tasks rather than analytics and machine learning. Additionally, robust data governance tools have enabled access while maintaining compliance, allowing for decentralized decision-making and data democratization.

The process of extracting ASCII files with fixed (csv/tsv) and variable (EDI/HL7) formats into a supported destination involves determining the input or load options based on the scenario, automating the transformation logic in dbt, and running the sample projects in Fivetran. The final target is a table called 'EDI_DOCUMENTS' that is configured in the transformation dropdown. To achieve this, one must create a pipeline that creates a table called EDIRAW in the schema EDITEST_AUG31, update the project to use a different source file, add new segments, and refine the UNION statements. The dbt project files need to be altered, including model files, segment mappings, and data files. VSCode can be used to run, modify, and test the dbt project in a standard developer environment without having to configure Fivetran. The samples provided are for demonstration purposes only, not meant for production, and further work could be performed to make use of other Fivetran and dbt related functionality.

Machine learning can be highly beneficial for businesses, but it requires a solid foundation of data infrastructure and mature data operations before pursuing projects that are open-ended, complicated, and risky. Before diving into machine learning, organizations should have a regular cadence of reports, governed and secured data, and widespread data-driven decision making. Once this foundation is in place, machine learning opportunities can be identified by considering the type of problem to be solved, such as pattern recognition, prediction, or automation, and applying machine learning flavors like unsupervised, supervised, or reinforcement learning. Practical examples of machine learning use cases include financial forecasting, personalization and recommendations, customer and product segmentation, marketing and sales forecasting, supply chain optimization, anomaly detection, and business process automation. These applications can be achieved through various machine learning flavors, including heuristics-based methods, and can have a significant impact on a business's productivity and competitiveness.

This blog post provides a comprehensive 30-day success guide for onboarding Fivetran at an organization. The guide covers four main goals: planning the path to success, connecting high-priority sources, becoming an expert in usage and pipeline activity, and reaching pro-level status. It includes tips on creating an implementation plan, managing connectors, understanding usage, forecasting costs, promoting a data-driven culture, and more. The guide aims to help users maximize their impact using Fivetran and become proficient in data automation.