October 2015 Summaries

The blog post from Snowplow discusses a simplified approach to orchestrating multi-stage ETL pipelines using Unix tools like Make and Cron, instead of more complex orchestration tools such as AWS Data Pipeline or Airflow. The author outlines how to define a Directed Acyclic Graph (DAG) using a Makefile to manage tasks and dependencies, and how to schedule these tasks using Cron for periodic execution. The post emphasizes the strengths of this approach, such as reduced complexity and easier troubleshooting, despite lacking advanced functionalities found in dedicated orchestration tools. It also covers how to handle job failures by modifying the Makefile to resume tasks from a point of failure, showcasing its practical applicability for prototyping and managing batch processing jobs.

Snowplow emphasizes the importance of accurately modeling event streams as businesses increasingly rely on them for insights, adopting a semantic model to prevent outdated assumptions from affecting data streams. The company's initial event grammar, based on linguistic slots like Subject, Verb, and Object, has evolved through extensive exposure to diverse event streams and customer interactions. This evolution led to a focus on entities, defined as relevant objects in events, and their interaction with time, acknowledging their static, infrequently, or frequently changing properties. Snowplow proposes "entity snapshotting," capturing the state of entities at the moment of an event, simplifying event capture and analysis. This approach updates the event grammar to focus on entity snapshots and verbs, facilitating representation in self-describing JSON, and aligns with their existing custom contexts. The revised model aims to be both simpler and more powerful, inviting community feedback to refine these concepts further.