May 2023 Summaries

The text discusses the challenges and pitfalls of implementing machine learning and artificial intelligence in organizations, highlighting the need for a strong foundation in data infrastructure, processes, and practices to support these technologies. To avoid common mistakes, such as overkill solutions or insufficient supporting infrastructure, organizations must identify specific problems that machine learning can solve and have mature enough data operations to support its use. The text also introduces two models for charting a progression to machine learning: the data maturity curve and the data hierarchy of needs, which describe an organization's orientation towards data and tools, processes, and capabilities with respect to data, respectively. By leveraging technological solutions to build a robust data infrastructure and developing a proactive and innovative orientation towards data, organizations can create a solid foundation for pursuing machine learning and realizing its potential.

Fivetran provides automated date integration for HL7 files, which are similar to EDI files but with an extra twist of nested records. To load HL7 data into a data warehouse, changes are required in the Fivetran S3 connector configuration, such as selecting the pipe field delimiter instead of asterisk and enabling the headerless files option. The data is then parsed using SQL views, which extract values from the pipe-delimited column using string functions specific to each CDP platform (e.g., Snowflake's `SPLIT_PART`, Databricks' `element_at`). The parsed data can be used to derive segment types, such as the MESSAGE_TYPE field, which contains a nested record structure. This approach allows for late binding solutions, which can parse data once in the CDP landing zone without rejecting it early on, making it ideal for analytic/innovation use-cases.

Replicating SAP data in the cloud offers numerous benefits, including creating a centralized repository for manufacturing data, eliminating data silos, and providing agile insights due to continuous operations. This can lead to better decision-making, heightened analytics, and more accurate reporting. However, replicating SAP data is challenging due to its complexity and large volumes of data stored within the system. Fivetran, a cloud-based ELT platform, automates this process by extracting data from SAP systems and loading it into a destination, making it easier and more efficient. By using Fivetran, organizations can significantly reduce the time and resources required to replicate SAP data, simplify the process of data replication, and accelerate the process of turning manufacturing data into insights.

Fivetran Activations introduces its first dbt package, dbt_census_utils, designed to streamline data activation processes by providing six macros that simplify data preparation for over 200 SaaS tools. These macros aid in Reverse ETL scenarios by handling tasks such as distinguishing personal from business emails, cleaning names for advertising platforms, and standardizing country codes. The package, developed with Stephen Ebrey of Sawtelle Analytics, aims to enhance data transformation and activation by resolving common issues like API record rejection and user-matching inaccuracies. The package supports multiple data warehouses, including Redshift, Snowflake, and BigQuery, and includes detailed documentation and integration tests for each macro. Users are encouraged to provide feedback and suggest new macros through GitHub or Slack, with further insights available in an upcoming webinar hosted by Stephen Ebrey.

Fivetran has launched its cloud region in Jakarta, Indonesia, expanding support for Google Cloud corporations and enabling low-latency data movement within local GCP environments. This launch supports Indonesian companies operating on Google Cloud, allowing them to facilitate faster transfer speeds and increased efficiency. The addition of this new region brings Fivetran's total number of supported regions across nine countries to 10. With its multi-cloud functionality, Fivetran provides global customers with scalability and flexibility to optimize their cloud provider usage and minimize data costs. The launch also supports the Business Critical platform for GCP users seeking high security and compliance features such as Google KMS and Private Service Connect integrations.

In today's complex data landscape, seamless data movement between sources and destinations is crucial for operational and analytical uses of data. Data movement supports various use cases such as business intelligence and decision support, predictive modeling, machine learning and AI, real-time exposure of data, business process automation and data-driven products. A high-performance system that can be deceptively complex to engineer is required to move data from sources to destinations. Key considerations include properly scoping and scaling an environment, ensuring availability, recovering from failures and rebuilding the system in response to changing data sources and business needs. Five critical features of modern data movement are automation, incremental updates, idempotence, schema drift handling, and pipeline and network performance. Automation ensures fully managed and automated data movement from the standpoint of the end user, while incremental updates provide real-time or streaming updates by identifying changes made to a previous state of the source. Idempotence ensures that if you apply the same data to a destination multiple times, you will get the same result, which is crucial for recovery from pipeline failures. Schema drift handling involves faithfully preserving original values and ensuring smooth passage from source to destination despite evolving application data models. Pipeline and network performance can be improved through algorithmic improvements, code optimization, architectural changes, and pipelining, with transformations being a key aspect of ELT architectures that alter raw data into usable structures called data models.

Fivetran is hosting an event at the Data + AI Summit 2023 in San Francisco, CA from June 26th-29th. Attendees can connect with Fivetran at their booth #315 to learn about their data integration platform and see a live demo. Additionally, Fivetran will be hosting speaking sessions, including "Centralizing Data Sources and Optimizing Data Movement: How Workday Standardized Company-Wide Reporting for Faster Insights With Fivetran and Databricks," and a happy hour event with other partners. Fivetran is also offering a chance to win a gift and make a donation to Girls Who Code when visiting the booth. Furthermore, attendees can book a meeting with the Fivetran team or schedule a live demo to learn more about the platform and its capabilities in integrating data from diverse sources into the Snowflake Data Cloud for analysis.

The modern data stack has made significant improvements in the quality and performance of analytics by providing automated data pipelines, data warehouses, transformation tools, and business intelligence platforms. A centralized and governed data layer is crucial for resolving challenges such as siloed reporting, conflicting findings, and sensitive data exposure. The same principles can be applied to modern data science architectures to address issues like lack of data governance, underdeveloped DataOps infrastructure, and regulatory compliance. By focusing on basic infrastructure and productionizing models, data scientists can improve their return on investment and bring machine learning projects into production using the same modern data stack used for business intelligence. This involves leveraging structured data in a data warehouse, utilizing governed data lakes, and applying analytics data storage technologies that support both relational and unstructured data.

Fivetran is a platform that can quickly and easily integrate data from various ASCII file formats, including EDI (Electronic Data Interchange) files, into a data warehouse. The platform uses a late binding model to parse the complex data in these files, breaking them down into distinct columns. This approach allows for fast ingestion of raw data and enables the use of massive parallel processing (MPP) cloud platforms to further process the data. By leveraging this solution, Fivetran can deliver accurate and complete data to analytics environments, including data with errors or inconsistencies, which can be valuable in understanding issues occurring within the data and originating systems. This approach also eliminates the need for 3rd party platforms or programming languages, making it a convenient option for organizations looking to integrate EDI files into their data pipelines.

Fivetran is hosting a booth at the upcoming Snowflake Summit 2023 in Las Vegas, where attendees can learn about their data integration platform and see it in action. The Fivetran team will be available to book meetings with potential customers, showcase their pre-built connectors that extract data from various sources, and provide live demos of their automated data pipelines. Additionally, they will host a happy hour event with other partners and offer VIP, bespoke dinners for Director-level executives. Fivetran is also partnering with Snowflake to provide a seamless data integration and analytics experience, and attendees can schedule a live demo or learn more about the partnership through various channels.

Fivetran Inc. is a company that offers an ELT (Extract, Load, Transform) platform designed to accelerate analytics by making data transformation simple, powerful and seamless within its own UI. The platform provides features such as prebuilt data models, version control, integrated scheduling, and CI/CD capabilities, all accessible through SQL. Fivetran's transformation layer integrates tightly with dbt, a leading SQL-based transformation tool, offering additional features that make data transformation faster and more powerful. With its capabilities, Fivetran enables fast, collaborative and transparent modeling, version control, branching and merging, and automated CI/CD. The platform also provides prebuilt data models, modularity, data lineage graphs, managing and scheduling transformations, and "fully integrated" scheduling, making it easier for teams to save time and keep data fresh. By implementing Fivetran's ELT platform, companies can create scalable and powerful transformations much faster, reducing the need for manual coding and improving business decision-making.

The concept of treating data assets like products is crucial for achieving data centralization and democratization. Data teams should adopt a product thinking mentality, focusing on creating products that solve practical problems for end-users of data assets. This involves understanding user needs, designing solutions iteratively, building minimum viable products (MVPs), launching internal data products, and assessing their impact through metrics such as user satisfaction and adoption rates. By applying this approach, data teams can develop high-impact projects, exercise product thinking, and ultimately create valuable data assets that support smooth operations within the organization.

When choosing a change data capture (CDC) solution, several key considerations come into play. The type of CDC is crucial, with log-based being the most accurate and efficient option for high-throughput data movement, trigger-based being less preferred due to its impact on database processing, and difference-based being suitable only for small data sets. The architecture of the CDC solution should be able to handle the volume of changes, with distributed systems offering better performance. A broad library of connectors is necessary to automate connections to various databases and applications, while managed services and support can eliminate time and expense associated with in-house development and maintenance. Security features such as enterprise-level security, encryption, anonymization, transparency, and granular role-based access control are also essential considerations. Ultimately, a CDC solution like Fivetran that offers log-based CDC, broad connectors, enterprise-grade security and governance features, and managed services can help organizations move their data quickly and securely into the cloud.

Fivetran's Composable CDP solution is revolutionizing the customer data platform landscape by offering a cloud-native approach that empowers organizations to build personalized experiences that drive business outcomes. With this modular and flexible system, companies can easily integrate data from multiple sources, gain valuable insights into their customers' behavior and preferences, and create targeted marketing campaigns that result in increased revenue. The solution is designed to help businesses stay agile and responsive to new requirements, delivering exceptional customer experiences that enhance engagement and drive business outcomes. Fivetran's Composable CDP solution is part of a broader ecosystem that includes Google Cloud BigQuery and Hightouch's automation platform, providing organizations with the tools they need to succeed in today's competitive digital landscape.

The text discusses building a data architecture for long-term success in the modern data industry. It highlights the importance of focusing on core business outcomes and avoiding vendor lock-in, instead opting for multi-cloud architectures that provide flexibility and scalability. The author also explores the debate between data warehouses, lakes, and lakehouses, emphasizing that each has its strengths and weaknesses and may not be suitable for every use case. The key to making an informed decision is to assess common scenarios and consider factors such as data governance, business intelligence, and performance requirements. Ultimately, a single platform that integrates well with upstream and downstream processes and can be programmatically controlled using APIs is often the best choice for supporting both structured and unstructured data types.

Fivetran has expanded its Business Critical product offering to support Google Cloud Platform (GCP), providing enterprise organizations with the most secure data movement solution now available on GCP. The platform, which was originally launched in 2021 and initially supported AWS platforms, offers end-to-end, fully-managed and automated data movement with high security features such as customer managed keys, private cloud connectivity, and enhanced data residency functionality. With Business Critical, Fivetran meets the high-security demands of enterprises with strict compliance requirements by offering premium security features like PCI DSS Level 1 compliance, VPN tunnels, SCIM/user provisioning, custom roles, column-level lineage, and integrations with catalog tools.

Marketers are increasingly facing challenges in creating personalized cross-channel experiences due to the lack of a unified customer view, despite having vast amounts of data from various sources. The Composable Customer Data Platform (CDP) offers a solution by utilizing the Snowflake Data Cloud and integrating tools like Fivetran for ELT, dbt for data transformation, and Snowplow for event collection, to create a more flexible and efficient marketing platform. Unlike traditional CDPs, which often lead to fragmented data silos, a Composable CDP uses existing data infrastructure to enable real-time customer engagement and personalization, avoiding duplicate data storage and ensuring consistent metrics. This approach not only reduces costs and speeds up data activation but also enhances privacy and security by keeping data within the organization’s infrastructure. Companies like Canva, Zip, and HOVER have successfully implemented Composable CDPs, resulting in improved customer segmentation and engagement. The future of marketing technology is moving towards composable solutions, leveraging platforms like Snowflake to bridge the gap between marketing and data teams and maximizing the potential of customer data.

The Fivetran Log Connector is a free tool that provides visibility into how users are using their Fivetran account, allowing for monitoring of instance performance, optimization of data integration costs, and control of data integration. The connector exposes log records containing sync statistics, user activities, and detailed sync information, which can be used to identify errors and troubleshoot issues. Fivetran uses the Log Connector to monitor its own demo account, which is shared with technical and sales teams and has high traffic. To prevent unnecessary infrastructure and warehouse costs, the team regularly cleans up the environment using automated scripts and tracks trends in their environment to determine where data volume comes from, what errors occur, and where those errors occur. The Log Connector is set up to write data to a specific location in the data warehouse, and role-based access controls are used to restrict access to the destination. The team uses this visibility to build a monitoring dashboard using Sigma, which provides insights into active connector count by destination, broken connectors by type, and other metrics.

The Fivetran Platform Connector provides users with visibility into their account usage, offering valuable log insights for large enterprises managing numerous users and pipelines. This free connector does not count towards monthly active rows (MAR). Users can monitor instance performance, identify optimizations, control data integration costs, and introduce new efficiencies using the information provided by the Platform Connector. Fivetran uses its own Platform Connector to automate monitoring of its demo account, which is shared with technical and sales teams for running product demos. The company has set up guidelines and policies for maintaining a clean Fivetran account, including regular cleanup of the environment using automated scripts.

As marketing agencies face uncertain economic conditions, it's essential to demonstrate the ROI of marketing spend. To efficiently uncover insights from client data without overextending resources, agencies can adopt modern data stack (MDS) practices, such as using fully managed ELT pipelines, cloud-based data warehouses or lakes, and business intelligence platforms. This allows data engineers to focus on more impactful aspects, like building a Composable Customer Data Platform (CDP), breaking down data silos, and providing actionable insights to clients. By centralizing marketing data and leveraging platforms like Fivetran, agencies can streamline processes, improve efficiency, and drive ROI for their clients. With flexible pricing based on usage, Fivetran enables agencies to grow and add more clients without increasing costs.

Immortal time' is a bias in the analysis of conversion rates, where a period of time is artificially extended due to events that prevent data from being collected, such as customers giving up on using a product or feature. This bias can lead to incorrect conclusions about the effectiveness of certain features or products. By recognizing and adjusting for this bias, analysts can get a more accurate picture of how different features or products impact conversion rates. In the context of Fivetran's analysis of transformation users versus non-users, immortal time was found to be a significant issue that needed to be addressed in order to draw meaningful conclusions about the impact of transformations on conversion rates.

Fivetran is a cloud-based data integration platform that offers a usage-based pricing model, allowing customers to only pay for what they use. To optimize their usage and costs, Fivetran provides six tips: adding all sources to take advantage of economies of scale, removing schemas that aren't bringing value to the business, using the Usage Estimator to plan usage and costs, evaluating usage as part of total cost of ownership, monitoring usage with alerts, and setting up new data notifications for new tables or columns. By following these tips, customers can streamline their data pipelines while managing their costs, focusing on gaining valuable insights from their data.

Kat Gardiner is a Marketing, Growth, and Analytics Consultant who frequently works with Series A startups to develop their data infrastructure. She advises early-stage organizations to start by understanding their business goals and creating a centralized data process. This involves identifying key data points that contribute to the organization's objectives, breaking down siloed data sources, and establishing a relational dashboard tree with level-specific metrics. Gardiner emphasizes the importance of automation in managing initial implementation with limited resources, highlighting tools like Fivetran as affordable solutions that can connect disparate data sources into one centralized destination. Her clients have experienced improved resource allocation, morale, and job satisfaction, as well as democratized access to data across the company.