July 2019 Summaries

This article discusses three common approaches to data analysis - ETL (Extract, Transform, Load), ELT (Extract, Load, Transform) and ELA (Extract, Load, Analyze). It highlights that while ETL and ELT are well-known concepts, they do not describe the most common agile analytics workflow. In practice, data teams often omit modeled transformations and perform extraction, loading and analysis (ELA). The article also compares these approaches by use of coded examples, primarily transformation and analysis. It concludes that both ELT and ELA have advantages and disadvantages, with the choice between them largely depending on factors such as cost, efficiency, quality and scalability concerns.

Fivetran simplifies loading data into Snowflake, offering secure connections for various sources. The architecture and security of Fivetran are discussed, along with a demonstration of their new transformation tool. Fivetran is designed by the needs of data analysts and efficiently replicates applications, databases, events, and files into high-performance cloud warehouses. It requires minimal maintenance and adjusts to source changes automatically. Snowflake, on the other hand, is a leading cloud-based data warehouse with unique architecture that delivers exceptional performance, concurrency, and simplicity. It allows multiple groups to access petabytes of data simultaneously, offering up to 200 times faster performance and 10 times lower cost compared to non-cloud solutions. Snowflake operates on structured and semi-structured data and follows a pay-as-you-go model with full management services.

Data becomes useful knowledge and products through several stages, including data gathering, extraction, processing, and use for business intelligence or powering products. The cloud plays a significant role in this process by offering accessible, cheap, performant, and scalable off-the-shelf solutions to IT infrastructure needs. Modern data stacks are hosted in the cloud, with many products and services being "cloud-native." Data sources can originate from various places such as sensor inputs, manual data entry, digital documents, or software triggers. Centralizing data is crucial for gaining a comprehensive view of an organization's operations and tracking individual entities across multiple applications. Processing the data involves transformations like data cleaning, summarizing, pivoting tables, and joining records from multiple sources together. Ultimately, the data is used to guide decisions at every level of an organization and power artificially intelligent products through analytics or business intelligence platforms.

Data plays a crucial role in guiding business decisions and powering AI products. The process of transforming raw data into useful knowledge involves several stages, including gathering, extracting, processing, and using the data. Modern data stacks are hosted in the cloud, which offers accessible, cheap, performant, and scalable off-the-shelf solutions to a range of IT infrastructure needs. The cloud enables organizations to easily scale their operations up and down without the need for proprietary IT infrastructure. Data sources can originate from various sources such as sensor inputs, manual data entry, digital documents, and software triggers. Organizations use a wide range of cloud applications to provide services like customer relationship management, payment processing, and enterprise resource planning. The centralization of data is achieved through data warehouses or data lakes, which store both structured and raw, unstructured data. Data pipeline tools are used for extraction and loading processes. Processing the data involves transformations such as cleaning, summarizing, and pivoting tables. Automating the ETL process requires careful coordination of different pieces of transformation software called orchestration. The rise of cloud technology has made the labor-intensive ETL approach obsolete, with ELT being a more efficient alternative. Data governance is an important concern that arises alongside centralizing and processing data to ensure proper usage, integrity, and security of data. Ultimately, the data from a data stack is meant to guide decisions at every level of an organization and power AI products through analytics or business intelligence platforms.

Product teams often face challenges in validating sales team requests due to lack of up-to-date usage data. Fivetran, BigQuery, Looker, and Brandwatch are tools that can help by providing real-time access to relevant data, enabling efficient collaboration between product and sales teams. These platforms offer seamless integration, high scalability, and valuable insights for driving business decisions.

Fivetran offers four data loading methods when a connector is not available for a specific source: file replication, webhooks, database applications, and cloud functions. File replication involves converting data into files that can connect to the destination using one of Fivetran's file replication services. Webhooks send data from applications that support them to the destination. Database applications require an underlying supported database for connection. Cloud functions allow users to code custom data connectors as extensions of Fivetran, while email connectors generate an email address for sending supported file types for ingestion.