From Batch to Streaming Data: Real Time Monitoring with Snowflake, Looker, and CockroachDB

The article discusses the transition from batch to streaming data for real-time monitoring using Snowflake, Looker, and CockroachDB. It explains how Cockroach Labs uses telemetry data collected internally on product usage to inform decisions about features and services. The company initially used weekly cron jobs and batch updates but switched to changefeeds for faster, more reliable data updates. Changefeeds watch a table or set of tables, pick up any changes to the underlying data, and emit those changes via a changefeed. This approach has added robustness and reliability to their process, allowing them to go from weekly to daily or near real-time metrics. The journey of their data involves using an AWS S3 bucket as a general-purpose data dump, loading the data into Snowflake using Snowpipe, and finally reading it in Looker for aggregation and displaying in dashboards.