March 2022 Summaries

Data teams face increasing challenges with the evolving complexity of the Modern Data Stack, requiring robust processes in data monitoring and observability to manage failure points and maintain trust in data. Observability, distinct from monitoring, involves the proactive identification and prevention of issues across data pipelines, while monitoring addresses known problems as they arise. Snowplow offers tools to enhance data pipeline observability, providing transparency and control over data with both open-source and managed SaaS options. By adopting a white box approach, Snowplow allows users to monitor pipeline health using metrics like latency and event volume through platforms like AWS CloudWatch and GCP Stackdriver. This setup enables teams to identify potential issues quickly, preventing data downtime by ensuring events conform to predefined schemas before entering data warehouses. Snowplow's design facilitates a deeper level of monitoring, helping users maintain reliable data systems by resolving validation errors preemptively and allowing immediate issue resolution based on alerts.

The article explores the concepts of observability and monitoring within data pipelines, using Snowplow BDP as a case study, emphasizing the importance of high-quality behavioral data. It explains how observability offers a high-level view of data health through metrics like event volume and latency, which help identify potential issues. In contrast, monitoring involves a more granular investigation into specific pipeline components to diagnose and resolve problems. Snowplow's approach focuses on creating a single, ultra-high-quality data table serving as a single source of truth, emphasizing the need for observability to prevent issues from escalating. The text details the use of monitoring dashboards and metrics, such as CPU utilization and stream latency, to maintain data pipeline efficiency and reliability, underscoring the significance of integrating observability into data workflows to mitigate the costs of data downtime and poor-quality data.

Snowplow has become an official Braze Alloys ISV Partner, aiming to enhance customer engagement through the integration of their platforms. Snowplow, a Behavioral Data Platform, and Braze, a customer engagement platform, collaborate to generate comprehensive behavioral data, which helps improve customer experiences by enabling personalized messaging. This partnership is designed to break down technological silos, allowing data and marketing teams to work collaboratively using shared behavioral data. The integration leverages the Modern Data Stack, where Snowplow sends structured data to a data warehouse, which is then operationalized into Braze via reverse ETL tools, leading to more effective campaigns. Additionally, the partnership uses Google Tag Manager Server-Side for enhanced data transformation visibility and reduced client load. A discussion featuring leaders from Auto Trader, Braze, Hightouch, and Snowplow highlighted the importance of the data warehouse in unifying engineering and marketing teams to better meet business needs.

Adopting a multiple data pipeline approach is essential for companies to comply with varying data privacy laws, particularly between the EU and the US, where different regulations like the GDPR and the CLOUD Act impose distinct requirements. The EU's General Data Protection Regulation (GDPR) grants citizens extensive rights over their data, mandating companies to protect personal identifiable information (PII) rigorously, as seen in cases where companies like Amazon and WhatsApp faced significant fines for non-compliance. In contrast, the US CLOUD Act allows government access to data without notifying the data subject, leading to conflicts exemplified by the Schrems II ruling, which invalidated the Privacy Shield framework for data transfers between the EU and the US. The French CNIL and Austrian DSB have enforced GDPR more strictly, highlighting the need for companies to adopt proactive data sovereignty strategies. Utilizing tools like Snowplow to create separate data pipelines for different regions can help mitigate risks associated with data transfers, ensuring compliance and allowing companies to manage data according to local laws while maintaining control over where and how data is stored and processed.