July 2019 Summaries

Moving to a modern cloud data warehouse such as Snowflake, BigQuery, or Redshift involves decisions about building data pipelines, specifically choosing between ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) methods. ETL traditionally processes data by transforming it before loading into a warehouse, which can be time-consuming and costly, especially with large datasets and the need for extensive infrastructure maintenance. In contrast, ELT leverages cloud capabilities by loading data first and transforming it within the warehouse, allowing for more flexible and efficient data handling. As businesses increasingly rely on numerous SaaS applications, managing real-time structured and unstructured data at scale has become crucial, making ELT an attractive option due to its ability to minimize infrastructure overheads and adapt to dynamic analytical needs. When selecting between ETL and ELT vendors, companies should consider factors such as flexibility in managing multiple data sources, compatibility with their chosen cloud data warehouse, and pricing structures, which may vary based on integration levels, data rows, or data volume.

Sigma has introduced a new feature that allows customers to upload spreadsheets directly into its platform, enabling users to integrate external CSV data with live data from their cloud data warehouses. This feature, which requires a Sigma Admin to enable write access to the database, facilitates the creation of a table in the connected warehouse from the uploaded CSV data, which Sigma then transforms into a worksheet. This worksheet can be shared and joined with other data sources within Sigma, providing a seamless analytics experience. The CSV upload process is user-friendly, with Sigma automatically detecting delimiter and escape characters, and offering options for customization. This feature enhances data security and governance by eliminating the need to extract company data from the warehouse, thus allowing business experts to merge their own data processes with organizational data efficiently.