December 2018 Summaries

The Snowplow pipeline, designed to prioritize data quality from the outset, uses predefined schemas to validate incoming data, with invalid data preserved as bad rows for further analysis. This process is crucial for debugging, especially in the Google Cloud Platform (GCP) environment where bad rows are streamed to Cloud Storage. Users can create external tables in BigQuery to monitor these bad rows in real-time and native tables for more detailed, cost-effective analysis. The guide explains how to handle bad rows by identifying validation failures, which may arise from non-Snowplow traffic or schema mismatches, and provides instructions for querying and decoding bad row data to diagnose issues. The process involves creating external and native tables, using SQL to limit queries and count errors, and decoding base64-encoded data to identify the source of validation errors, ultimately allowing users to refine their data tracking and schemas for improved data quality.

Snowplow Analytics has partnered with Google Cloud Platform (GCP) to enhance the processing of highly detailed and structured data, leveraging GCP's BigQuery for efficient large-scale analytics. BigQuery's capabilities, such as support for nested data structures, streaming inserts, and seamless integration with Google marketing services, make it an ideal platform for storing and querying massive datasets, including over 100 million daily events tracked by Snowplow users. This integration allows real-time data availability, enabling fast data-driven decision-making across marketing and product development. The combination of Snowplow's data granularity and BigQuery's robust infrastructure ensures that users can easily unify diverse data sources into a high-quality dataset, benefiting from GCP's affordability and performance. By incorporating tools like TensorFlow for machine learning and Data Studio for visualization, Snowplow and GCP together provide a comprehensive suite for managing and analyzing data effectively, allowing companies to optimize their marketing strategies and operational efficiency.