July 2020 Summaries

As companies increasingly prioritize the ownership and control of their data infrastructure, the debate between building versus buying data pipelines has become prominent, with many opting to invest in building their own systems for greater flexibility and customization. This trend is driven by the desire to have full control over data, which is seen as a strategic asset, and to tailor infrastructure to specific business needs without being constrained by third-party vendors. However, building a data pipeline involves significant challenges, including the need for ongoing maintenance, resource allocation, and strategic planning, which can be complex and resource-intensive. Despite these challenges, some organizations have successfully implemented self-built pipelines, enjoying benefits such as enhanced control and ownership. For those seeking a middle ground, solutions like Snowplow BDP offer a fully managed service that allows companies to maintain ownership and flexibility while outsourcing the maintenance and management of their data pipeline infrastructure, combining the best aspects of both self-built and purchased solutions.

Data latency, a crucial metric for data teams, measures the time it takes for data to become accessible in a database after an event occurs, and is vital for executing real-time or near real-time data use cases such as fraud detection or recommendation engines. While some applications, like quarterly sales reports, can tolerate higher latency, others benefit significantly from rapid data availability, enhancing commercial value by enabling faster actions on data. Despite its importance, obtaining latency metrics is challenging as most data platforms don't provide easy access to this information. Snowplow addresses these challenges by updating its microservices to allow customers to measure data latency accurately and consistently, enabling them to track the performance of data pipelines and understand the speed at which data is loaded into storage. By providing detailed latency metrics, Snowplow enhances transparency and confidence in data products, publishing the data to platforms like Google Cloud's Operations and Cloudwatch, which aids in monitoring and improving the health of data pipelines.

Version 0.5.1 of BigQuery Loader introduces a refined latency metric that measures the time taken for data to travel from the collector to its ingestion in BigQuery, utilizing Google Cloud Platform's Logging service for easy access and integration with monitoring tools like Prometheus and Grafana. The metric is calculated by sampling data every second during loading and determining latency based on the timestamps at the collector and just before loading to BigQuery. Insights Customers can access this metric directly in the Google Cloud Console's Monitoring UI, while open source users must create a custom logs-based metric using the Logs Viewer. The aggregation settings in Google Cloud Console or Grafana can be adjusted to enhance data granularity, with recommendations to align data at one-minute intervals and use the mean as the aggregator. The release notes for versions 0.5.0 and 0.5.1 on GitHub detail the changes, with automatic upgrades for Insights Customers and specific guidance available for open source users upgrading from older versions.