September 2024 Summaries

Snowplow has launched its new Video & Media Data App, designed to help businesses optimize content, enhance audience retention, and evaluate the effectiveness of media marketing strategies by providing comprehensive insights into video, streaming, and audio engagement. This application builds on Snowplow's Media SDKs and Media dbt package, offering interactive dashboards that allow data and marketing teams to visualize key metrics such as total watch time, audience retention, and advertising effectiveness. With the increasing reliance on video as a marketing tool, the app addresses the challenge many companies face in capturing and analyzing detailed engagement data, further enhanced by visualizations and dashboards from the previous Media Analytics Accelerator. The app features three main dashboards—Overview, Content, and In-Media Ads—that provide insights into overall media performance, individual content engagement, and the efficacy of media advertising, ultimately enabling businesses to better understand and optimize their media strategies.

Big Data LDN 2024 highlighted the importance of high-quality, proprietary data as the foundation of AI initiatives, emphasizing the adage "garbage in, garbage out" for AI applications. The event underscored the significance of data flywheels, which create competitive advantages by continuously refining AI systems through self-reinforcing cycles of proprietary data. Sessions also explored modern data lake formats, stressing the need for scalable and cost-effective data storage solutions to support AI workloads. Building a data-driven culture within organizations was deemed crucial for leveraging data and AI technologies effectively, with a focus on integrating data into the operational mindset. The rise of AI agents as autonomous systems capable of complex decision-making was discussed, positioning them as essential for maintaining a competitive edge, despite challenges in integration, user-friendliness, and ethical considerations. Overall, the event conveyed that prioritizing data quality, culture, and innovative approaches is vital in the evolving AI landscape.

Implementing a real-time reporting system using AWS Lambda, DynamoDB, and Snowplow's enriched event stream enables immediate insights for applications such as tracking active players in a game level. AWS Lambda and DynamoDB provide a serverless, low-latency infrastructure that automatically scales to handle fluctuating event volumes and offers cost efficiency by charging only for the compute and storage used. The setup involves configuring a Lambda function to process events from a Kinesis stream and updating a PlayerState table in DynamoDB to store each player's current level and timestamp. The system also includes functions to increment or decrement player counts in the LevelState table as players enter or exit levels, and a pruning function to handle inactive players by periodically cleaning up records. This architecture allows for the creation of a robust real-time reporting system with minimal latency, with further optimizations to be discussed in a subsequent part.

Snowplow's non-lossy pipeline architecture is distinguished by its ability to preserve malformed or invalid events as "bad rows" rather than discarding them, providing detailed error messages that help identify data quality issues. This approach is valuable for data teams rolling out new tracking systems or monitoring pipeline integrity over time. The tutorial outlines the process of querying and debugging these bad rows using Amazon Athena, a serverless SQL-based query service, and is designed for data engineers, architects, and analytics professionals familiar with AWS, Snowplow, and SQL. By creating an Athena table that maps to the bad row structure in S3, users can analyze error frequencies, preview types of errors, filter non-critical issues, and isolate specific recurring problems, which helps improve tracking reliability. The guide also suggests visualizing trends and error categories using tools like Amazon QuickSight or Redash and discusses recovery options such as Hadoop Event Recovery and custom JavaScript functions. Ultimately, Snowplow's method enhances data transparency and supports building more trustworthy data pipelines, allowing teams to maintain a reliable and high-integrity event stream.

User journey visualization, particularly through Sankey diagrams, is a compelling method for understanding user behavior on websites or apps, and Snowplow's event data facilitates creating these visualizations by offering session-level and page-level events with timestamped navigation sequences. Unlike tools like Google Analytics, Snowplow provides granular, first-party data that allows for custom, flexible, and privacy-respecting journey mapping. To generate a Sankey diagram, data must be structured into pairs of sequential steps with flow counts, which can be achieved using tools like dbt, SQL, or Spark. While Snowplow does not offer built-in Sankey visualizations, it integrates with various BI tools and visualization libraries, such as Looker, Tableau, D3.js, and Plotly, enabling users to tailor the visualization to their needs. Best practices for creating effective Sankey diagrams include limiting unique steps, using meaningful labels, segmenting audiences, and pre-aggregating data to enhance readability and performance.

Customer Data Platforms (CDPs) and Customer Data Infrastructure (CDI) represent two distinct approaches to managing customer data, with significant implications for organizations aiming to leverage data-driven strategies. Packaged CDPs offer an all-in-one solution that integrates data storage, processing, and analysis, primarily benefiting marketing and growth functions by facilitating customer engagement and optimizing ad costs. In contrast, CDI is a flexible, modular system that operates within an organization's data warehouse or lakehouse, providing greater control, data quality, and integration with advanced platforms like Snowflake or Databricks. As businesses increasingly seek to harness AI and BI-ready data, the rise of CDI reflects a shift towards more adaptable, future-proof architectures that accommodate complex workflows and stakeholder needs beyond marketing, such as Product and Customer Support. Organizations must carefully assess their data infrastructure, privacy requirements, and long-term costs to determine the best approach, with CDI often favored by larger enterprises with sophisticated data strategies.

The introduction of the General Data Protection Regulation (GDPR) has imposed significant changes on how organizations manage personal data, particularly with the "right to be forgotten" which mandates the deletion of personal data upon request. For companies using Snowplow, its structured event data model facilitates this process, particularly for deleting user data from Amazon Redshift. The guide outlines a step-by-step process, beginning with identifying and isolating the events associated with a specific user, followed by deleting these from both derived tables and the main events table, and concluding with post-deletion cleanup such as vacuuming and logging the time range of deleted events. Additionally, it emphasizes the importance of reviewing snapshot retention policies and maintaining best practices for GDPR compliance, such as using consistent identifiers, automating the deletion process, and regularly auditing and logging deletions. Ultimately, Snowplow's model, by isolating event metadata and separating enriched data, allows for systematic and confident user data deletions, aligning with privacy-by-design principles.

As the trend towards containerized, cloud-native infrastructure grows, many Snowplow users are exploring the deployment of their entire data pipeline on Kubernetes, leveraging platforms like AWS EKS and GKE. The Snowplow community confirms that running the full pipeline, including collectors, enrichers, loaders, and real-time processing infrastructure, is feasible on Kubernetes, despite some complexities and the need for custom engineering. Community resources like Helm charts and YAML files provide a starting point, though they often require customization, especially for handling IAM roles, logging, and metrics. Challenges such as IAM role binding issues, lack of Kafka support in some loaders, and the absence of unified Helm charts necessitate user intervention and adaptation. Best practices include defining the target stack, utilizing community charts, and following AWS IAM role practices, with ongoing community contributions enhancing the Kubernetes deployment experience for Snowplow users.

In response to the limitations of Amazon Redshift for large-scale data transformation, particularly in handling real-time processing and complex transformations, Snowplow transitioned to using Apache Spark. Redshift, while efficient for OLAP-style queries, encountered performance bottlenecks and lacked flexibility for mutable data and custom logic. Apache Spark, a distributed computing engine, provided a scalable solution with its ability to handle large-scale transformations, support for Python, Scala, and SQL, and seamless integration with cloud storage like S3. The transition involved redefining data models, loading raw data in Spark, performing transformations, and writing results to storage solutions such as S3, Snowflake, or BigQuery. Running Spark jobs on Amazon EMR was recommended for scale, with tips for migration including starting small, using version control, and monitoring performance. The shift to Spark enabled more expressive logic, improved performance, and reduced operational complexity, facilitating faster iterations and deeper insights from Snowplow's event data.

Marketers often face challenges in accurately attributing conversions to the correct marketing channels, as a study by Rakuten shows that over a quarter of their budget is wasted on ineffective strategies due to the complexity of customer journeys and the overlapping claims of different channels. Effective marketing attribution is essential for optimizing advertising spend and understanding the true return on investment from various campaigns. Single-touch attribution models, such as First-Touch and Last-Touch, are simple but often overlook the contributions of multiple interactions, while multi-touch models, including rule-based and algorithmic approaches, provide a more nuanced view by distributing credit across various touchpoints. Algorithmic models, like the Data-Driven (Shapley) model, offer an objective analysis based on statistical patterns but require significant data and computational resources. The article suggests starting with simple models and progressing to more complex ones as data and resources allow, emphasizing the importance of experimentation and adjustment in marketing strategies. Tools like Snowplow can assist in generating, managing, and modeling data for effective attribution, offering solutions that include tracking, data modeling, and visualization to help marketers better understand where their advertising spend is most effective.

Mastering foundational SQL queries is crucial for extracting actionable insights from Snowplow data, which is rich in event-level information. Essential SQL recipes for web data analysis include counting page views by URL to assess user engagement, calculating average session duration to understand content interaction time, and identifying top referrers to track sources of traffic. Additionally, calculating conversion rates helps measure key actions like sign-ups or purchases, and tracking users across sessions allows for deeper behavioral analysis. These foundational queries enable data engineers and analysts to quickly extract insights and form the basis for developing more complex data models, with further advanced SQL recipes to be explored in future discussions.

OpenTelemetry has emerged as a standard for collecting and transmitting telemetry data across platforms, and although Snowplow does not natively support OpenTelemetry, it can integrate with it through custom schemas and webhooks. By leveraging Snowplow's flexible data pipeline architecture and Iglu schema repository, engineering teams can ingest OpenTelemetry data and process it through enrichment pipelines for downstream analysis. This integration enables the combination of event data with tracing data, offering a comprehensive view of user behavior and system performance, and maintains data consistency by using standardized telemetry formats. Potential use cases include application performance monitoring, error tracking, debugging, and user journey mapping, utilizing tools such as Looker for dashboards, Grafana Tempo for trace visualization, and dbt with Redshift for deeper analysis. Overall, while Snowplow lacks native OpenTelemetry support, its open architecture allows for powerful integrations that can enhance observability strategies.

The new Marketing Attribution Data App launched by Snowplow aims to help companies accurately measure and optimize their marketing performance by leveraging first-party behavioral data from their own cloud data warehouse or lake, avoiding reliance on third-party cookies and analytics tools. This app addresses the inefficiencies in current marketing attribution by offering multi-touch attribution models that provide a comprehensive and unbiased view of campaign effectiveness across multiple channels and touchpoints. It allows marketers to improve ROI by understanding marketing performance beyond simplistic first and last-touch models, using sophisticated multi-touch approaches, including linear and positional models. Snowplow's app also offers features such as dashboards for essential marketing metrics, customer journey path analysis, and model comparisons, while ensuring compliance, data ownership, and privacy through its "glass box" philosophy and advanced data anonymization techniques. By enabling marketers, analysts, and data scientists to access rich, AI-ready customer data, the app empowers them to optimize marketing strategies more effectively, potentially reducing third-party marketing technology costs and enhancing overall marketing efficiency.